{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configuring Rendering Lighting\n",
    "For instructions on how to run these tutorial notebooks, please see the [index](./index.ipynb).\n",
    "\n",
    "### tl;dr\n",
    "\n",
    "This tutorial is about controlling the illumination in your rendered\n",
    "scenes. The goal is to give you proper intuition and techniques for\n",
    "efficiently and effectively lighting your scenarios. The goal is to\n",
    "end up with renderings like this (rendered in the\n",
    "[final section](#Putting-it-all-together) of this tutorial):\n",
    "\n",
    "![PBR-rendered robot](https://drake.mit.edu/doxygen_cxx/final_robot.jpg)\n",
    "**Figure:** Example of a rendering using `RenderEngineVtk` PBR rendering.\n",
    "\n",
    "The discussion is limited to a single rendering technology: physically-\n",
    "based rendering (PBR). This is available when using `RenderEngineVtk`.\n",
    "`RenderEngineVtk` also has a legacy Phong illumination model (also\n",
    "used by `RenderEngineGl`). This tutorial will not discuss the Phong\n",
    "illumination model. If curious, we recommend further reading on\n",
    "[Wikipedia](https://en.wikipedia.org/wiki/Phong_reflection_model).\n",
    "\n",
    "**Tutorial Overview**\n",
    "\n",
    "  - [Overview of rendering model (PBR)](#what_is_pbr)\n",
    "  - [Tutorial methodology and infrastructure](#infrastructure)\n",
    "  - [Direct lighting](#direct_lighting)\n",
    "    - [Shadows](#shadows)\n",
    "  - [Materials](#pbr_materials)\n",
    "  - [Environment maps](#environment_maps)\n",
    "    - [Tone mapping and exposure](#exposure)\n",
    "  - [Total illumination](#total-illumination)\n",
    "  - [Putting it all together](#Putting-it-all-together)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"what_is_pbr\"></a>\n",
    "## Physically-based Rendering (PBR)\n",
    "\n",
    "An illumination model is an approximation of the physics behind the\n",
    "physical phenomena underlying vision -- electromagnetic radiation\n",
    "strikes materials and reflects towards a photo-sensitive medium (e.g.,\n",
    "film, CCDs, human eyes, etc.) The so-called \"physically-based rendering\"\n",
    "illumination model is a modern model that hews more closely to actual\n",
    "physics (in comparison with older, classical illumination models).\n",
    "While it is truer to reality, it is still an approximation. There\n",
    "will inevitably be differences between renderings and reality.\n",
    "\n",
    "### What makes it physical?\n",
    "\n",
    "PBR's primary distinction is that it returns to the underlying physics of\n",
    "light:\n",
    "\n",
    "  - It's based on a statistical model of how light interacts with materials at a microscopic level.\n",
    "  - It conserves energy.\n",
    "\n",
    "PBR provides concrete benefits:\n",
    "\n",
    "  - It allows content authors to create models based on an object's\n",
    "    actual properties without concern about how the objects get used;\n",
    "    the correctness of the material doesn't depend on the lighting.\n",
    "  - Simply put, it produces more realistic images.\n",
    "  \n",
    "To effectively light a scene, you need to understand the properties\n",
    "of the available light sources _and_ how materials respond to those\n",
    "light sources. For example, if you have an object painted with\n",
    "[Vantablack](https://en.wikipedia.org/wiki/Vantablack) shining\n",
    "more light on the object will not appreciably change your final\n",
    "image. This tutorial provides a high-level discussion of the material\n",
    "properties and light sources available to you. If you're interested\n",
    "about the details of PBR, you might try\n",
    "[this tutorial](https://learnopengl.com/PBR/Theory) or \n",
    "[this documentation of a particular implementation](https://google.github.io/filament/Filament.html).\n",
    "\n",
    "### Enabling PBR in RenderEngineVtk\n",
    "\n",
    "The discussions below will be interleaved with interactive examples\n",
    "where you can explore the parameters discussed. We'll simply start\n",
    "with how to enable PBR illumination.\n",
    "\n",
    "`RenderEngineVtk` defaults to the Phong illumination model for\n",
    "historical reasons. This default will change in the future. In the\n",
    "meantime, there are three things that will change a `RenderEngineVtk`\n",
    "instance from Phong to PBR:\n",
    "\n",
    "  - Explicitly request it in the [`RenderEngineVtkParams`](https://drake.mit.edu/doxygen_cxx/structdrake_1_1geometry_1_1_render_engine_vtk_params.html)\n",
    "    - Set the `force_to_pbr` flag to `true`.\n",
    "  - Add an [environment map](#environment_maps).\n",
    "  - Add at least one glTF file (Drake's preferred visual geometry\n",
    "    representation) to the `RenderEngineVtk` instance. More specifically, add a\n",
    "    `drake::geometry::Mesh` with `drake::geometry::PerceptionProperties`\n",
    "    compatible with your `RenderEngineVtk` instance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"infrastructure\"></a>\n",
    "## Tutorial methodology and infrastructure\n",
    "\n",
    "### Methodology\n",
    "\n",
    "As we discuss the details of PBR and Drake, we'll do it in a context\n",
    "in which you can explore the parameter space.\n",
    "\n",
    "In addition to the code blocks in which you can modify parameters and\n",
    "re-evaluate, we've given you the ability to move the camera and\n",
    "re-render the image. We'll do that via an instance of meshcat\n",
    "visualizer. By moving the camera in the visualization browser window,\n",
    "the camera in this session will update its pose to match. It *does*\n",
    "require explicitly re-rendering.\n",
    "\n",
    "Each render will print the command for setting the camera pose. If you\n",
    "change the camera to something you like and would like to consistently\n",
    "reproduce it, you can copy and paste the command into the render\n",
    "window.\n",
    "\n",
    "The pattern you'll see over and over again is a pair of code blocks.\n",
    "The first will set up the scene -- with particular emphasis on the\n",
    "parameters being illustrated. It also sets a curated camera pose so\n",
    "that the first rendering you do will show what the tutorial needs to\n",
    "show.\n",
    "\n",
    "The second code block will simply perform the rendering (and contain\n",
    "the rendered result). The first evaluation will use the camera pose\n",
    "defined in the first block. Subsequent executions of the rendering\n",
    "block will use the pose from the meshcat session.\n",
    "\n",
    "To streamline the first code block, we'll introduce some infrastructure\n",
    "to admit compact declarations of scenes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Infrastructure\n",
    "\n",
    "The necessary imports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import copy\n",
    "import json\n",
    "import logging\n",
    "import math\n",
    "import os\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "from pydrake.common import MemoryFile\n",
    "from pydrake.common.value import Value\n",
    "from pydrake.geometry import (\n",
    "    Box,\n",
    "    ClippingRange,\n",
    "    ColorRenderCamera,\n",
    "    EnvironmentMap,\n",
    "    EquirectangularMap,\n",
    "    FramePoseVector,\n",
    "    GeometryFrame,\n",
    "    GeometryId,\n",
    "    GeometryInstance,\n",
    "    InMemoryMesh,\n",
    "    IllustrationProperties,\n",
    "    LightParameter,\n",
    "    MakeRenderEngineVtk,\n",
    "    Mesh,\n",
    "    Meshcat,\n",
    "    MeshcatVisualizer,\n",
    "    PerceptionProperties,\n",
    "    QueryObject,\n",
    "    RenderCameraCore,\n",
    "    RenderEngineVtkParams,\n",
    "    Rgba,\n",
    "    SceneGraph,\n",
    "    Sphere,\n",
    "    StartMeshcat,\n",
    ")\n",
    "from pydrake.math import RigidTransform, RotationMatrix\n",
    "from pydrake.multibody.parsing import PackageMap\n",
    "from pydrake.systems.framework import DiagramBuilder, LeafSystem\n",
    "from pydrake.systems.sensors import (\n",
    "    CameraInfo,\n",
    "    ImageIo,\n",
    "    ImageRgba8U,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll create a special system that will track the camera\n",
    "position in the meshcat session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CameraController(LeafSystem):\n",
    "    def __init__(self, scene_graph, meshcat):\n",
    "        \"\"\"The single output port provides a RigidTransform defining the\n",
    "        pose of the camera body in the world frame: X_WC. The value is defined\n",
    "        by polling the given Meshcat instance. If the meshcat url includes the\n",
    "        argument `tracked_camera=on` we'll get the pose of the visualizer's\n",
    "        camera. Otherwise, we simply return some generic camera position.\n",
    "        \"\"\"\n",
    "        LeafSystem.__init__(self)\n",
    "        self._meshcat = meshcat\n",
    "        self._source_id = scene_graph.RegisterSource(\"camera_controller\")\n",
    "        self._frame_id = scene_graph.RegisterFrame(\n",
    "            source_id=self._source_id, frame=GeometryFrame(\"camera\"))\n",
    "        output_cls = Value[FramePoseVector]\n",
    "        self.DeclareAbstractOutputPort(\"camera_pose\",\n",
    "                                       lambda: output_cls(),\n",
    "                                       self.CalcPose)\n",
    "        # Looking down from above.\n",
    "        self._default_X_WC = RigidTransform(\n",
    "            R=RotationMatrix.MakeXRotation(math.pi), p=[0, 0, 5])\n",
    "        self._X_WC = RigidTransform.Identity()\n",
    "\n",
    "    def frame_id(self):\n",
    "        return self._frame_id\n",
    "\n",
    "    def source_id(self):\n",
    "        return self._source_id\n",
    "\n",
    "    def set_camera_pose(self, pose_in_world=None):\n",
    "        \"\"\"Sets the override camera pose.\"\"\"\n",
    "        self._X_WC = pose_in_world\n",
    "\n",
    "    def CalcPose(self, context, pose_vector):\n",
    "        X_WC = self._X_WC\n",
    "        if X_WC is None:\n",
    "            X_WC = self._meshcat.GetTrackedCameraPose()\n",
    "            if X_WC is None:\n",
    "                X_WC = self._default_X_WC\n",
    "        def to_string(v):\n",
    "            return f\"[{', '.join([f'{vi:.5f}' for vi in v])}]\"\n",
    "        print(f\"comparator.pose_camera_once(camera_in_world=\"\n",
    "              f\"{to_string(X_WC.translation())},\\n\"\n",
    "              f\"                            target_in_world=\"\n",
    "              f\"{to_string(X_WC.rotation().matrix()[:, 2] + X_WC.translation())})\")\n",
    "        poses = FramePoseVector()\n",
    "        poses.set_value(id=self._frame_id, value=X_WC)\n",
    "        pose_vector.set_value(poses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll create the `RenderComparator` class. This class will\n",
    "allow us to instantiate multiple renderers for a single scene.\n",
    "When we render from the comparator, each of the render engines\n",
    "will render a color image, and the comparator will display the\n",
    "array of images.\n",
    "\n",
    "It also gives us short-cuts for populating the scene with simple\n",
    "primitives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RenderComparator:\n",
    "    \"\"\"Configures a diagram that can produce render comparisons between\n",
    "    multiple engines. The camera poses will be driven by a provided Meshcat\n",
    "    instance. Each time the rendering is made, the camera pose is pulled from\n",
    "    `meshcat`.\n",
    "    \"\"\"\n",
    "    def __init__(self, meshcat, builder = None):\n",
    "        \"\"\"Sets up the comparator to broadcast to the given `meshcat` instance.\n",
    "        If `builder` is not None, it must contain a SceneGraph instance\n",
    "        with the name \"scene_graph\".\n",
    "        \"\"\"\n",
    "        self._meshcat = meshcat\n",
    "        meshcat.Delete(\"/drake\")\n",
    "        if builder is None:\n",
    "            builder = DiagramBuilder()\n",
    "            self._scene_graph = builder.AddNamedSystem(\"scene_graph\", SceneGraph())\n",
    "        else:\n",
    "            self._scene_graph = builder.GetSubsystemByName(\"scene_graph\")\n",
    "        # Note: we'll be changing render engines a lot. So, rather than declare\n",
    "        # them in SceneGraph's model, we'll simply wait and add them to the\n",
    "        # context.\n",
    "        self._source_id = self._scene_graph.RegisterSource(\"main\")\n",
    "\n",
    "        MeshcatVisualizer.AddToBuilder(builder, self._scene_graph,\n",
    "                                       self._meshcat)\n",
    "\n",
    "        self._cam_controller = builder.AddSystem(\n",
    "            CameraController(self._scene_graph, self._meshcat))\n",
    "        builder.Connect(self._cam_controller.GetOutputPort(\"camera_pose\"),\n",
    "                        self._scene_graph.get_source_pose_port(\n",
    "                            self._cam_controller.source_id()))\n",
    "        builder.ExportOutput(self._scene_graph.get_query_output_port(),\n",
    "                            \"query_object\")\n",
    "\n",
    "        self._diagram = builder.Build()\n",
    "        self._context = self._diagram.CreateDefaultContext()\n",
    "        # Make sure MeshVisualizer publishes content to meshcat.\n",
    "        self._diagram.ForcedPublish(self._context)\n",
    "\n",
    "        self._sg_context = self._scene_graph.GetMyContextFromRoot(self._context)\n",
    "\n",
    "        self._cameras = []\n",
    "\n",
    "        # The contents of sphere.gltf -- we'll use it to create variations of a\n",
    "        # PBR sphere.\n",
    "        self._ball_count = 0\n",
    "        package_map = PackageMap()\n",
    "        self._gltf_supporting_files = {\"sphere.bin\": package_map.ResolveUrl(\n",
    "                \"package://drake_models/tutorials/models/assets/sphere.bin\")}\n",
    "        gltf_path = package_map.ResolveUrl(\n",
    "                \"package://drake_models/tutorials/models/assets/sphere.gltf\")\n",
    "        with open(gltf_path) as f:\n",
    "            self._ball_contents = json.load(f)\n",
    "\n",
    "        self._camera_info = {\n",
    "            # This matches the default Meshcat vertical field of view.\n",
    "            \"fov_y\": 75 / 180 * math.pi,\n",
    "            \"width\": 1024,\n",
    "            \"height\": 768\n",
    "        }\n",
    "\n",
    "    def set_camera_parameters(self, params):\n",
    "        \"\"\"Updates the camera parameters to include the values in `params`.\"\"\"\n",
    "        self._camera_info = self._camera_info | params\n",
    "\n",
    "    def _make_camera(self, renderer_name):\n",
    "        \"\"\"Makes a camera referencing the given `renderer_name`.\"\"\"\n",
    "         # This matches the default Meshcat vertical field of view.\n",
    "        fov_y = 75 / 180 * math.pi\n",
    "        core = RenderCameraCore(\n",
    "            renderer_name=renderer_name,\n",
    "            intrinsics=CameraInfo(**self._camera_info),\n",
    "            clipping=ClippingRange(0.1, 100.0),\n",
    "            X_BS=RigidTransform())\n",
    "        return ColorRenderCamera(core=core)\n",
    "\n",
    "    def pose_camera_once(self,\n",
    "                         camera_in_world,\n",
    "                         target_in_world=np.array((0, 0, 0)),\n",
    "                         up_in_world=np.array((0, 0, 1))):\n",
    "        \"\"\"Computes a pose for the camera located at `camera_in_world` (p_WC),\n",
    "        looking at `target_in_world` (p_WT) with the camera's up direction as\n",
    "        close to `up_in_world` (up_W) as possible. Passes it to\n",
    "        set_override_camera_pose(), so this pose will only be valid for the\n",
    "        next invocation of render().\n",
    "\n",
    "        The camera looks in the v_CT_W direction. v_CT should not point in the\n",
    "        same direction as up_W.\n",
    "        \"\"\"\n",
    "        p_WC = np.array(camera_in_world)\n",
    "        p_WT = np.array(target_in_world)\n",
    "        up_W = np.array(up_in_world)\n",
    "\n",
    "        def normalized(v):\n",
    "            return v / np.linalg.norm(v)\n",
    "\n",
    "        Cz_W = normalized(p_WT - p_WC)\n",
    "        # Note: -Cy points goes up in the image.\n",
    "        Cx_W = normalized(np.cross(-up_W, Cz_W))\n",
    "        Cy_W = normalized(np.cross(Cz_W, Cx_W))\n",
    "\n",
    "        X_WC = RigidTransform(RotationMatrix(\n",
    "                                  R=np.column_stack((Cx_W, Cy_W, Cz_W))),\n",
    "                              p_WC)\n",
    "        self.set_override_camera_pose(X_WC)\n",
    "\n",
    "    def set_override_camera_pose(self, pose_in_world):\n",
    "        self._cam_controller.set_camera_pose(pose_in_world)\n",
    "\n",
    "    def visualize(self):\n",
    "        \"\"\"Make sure MeshVisualizer publishes content to meshcat. This should be\n",
    "        called after adding geometry to the scene.\n",
    "        \"\"\"\n",
    "        self._diagram.ForcedPublish(self._context)\n",
    "\n",
    "    @staticmethod\n",
    "    def _assign_materials(geometry, diffuse_rgba):\n",
    "        def set_diffuse_maybe(props):\n",
    "            if diffuse_rgba is not None:\n",
    "                props.AddProperty(\"phong\", \"diffuse\", diffuse_rgba)\n",
    "\n",
    "        illustration = geometry.mutable_illustration_properties()\n",
    "        if illustration is None:\n",
    "            illustration = IllustrationProperties()\n",
    "            set_diffuse_maybe(illustration)\n",
    "            geometry.set_illustration_properties(illustration)\n",
    "        else:\n",
    "            illustration.UpdateProperty(\"phong\", \"diffuse\", diffuse_rgba)\n",
    "\n",
    "        perception = geometry.mutable_perception_properties()\n",
    "        if perception is None:\n",
    "            perception = PerceptionProperties()\n",
    "            set_diffuse_maybe(perception)\n",
    "            geometry.set_perception_properties(perception)\n",
    "        else:\n",
    "            perception.UpdateProperty(\"phong\", \"diffuse\", diffuse_rgba)\n",
    "\n",
    "    def add_ground(self):\n",
    "        \"\"\"Adds a grey ground.\"\"\"\n",
    "        ground = GeometryInstance(X_PG=RigidTransform([0, 0, -0.5]),\n",
    "                                  shape=Box(20, 20, 1), name=\"ground\")\n",
    "        self._add_geometry(ground, Rgba(0.4, 0.4, 0.4))\n",
    "\n",
    "    def _get_sphere_shape(self, pbr_material: dict):\n",
    "        \"\"\"Returns a Mesh shape derived from sphere.gltf whose pbr materials is\n",
    "        updated to reflect the material definition in `pbr_material`.\"\"\"\n",
    "        gltf_json = copy.deepcopy(self._ball_contents)\n",
    "        gltf_json[\"materials\"][0][\"pbrMetallicRoughness\"] = (\n",
    "            gltf_json[\"materials\"][0][\"pbrMetallicRoughness\"] | pbr_material)\n",
    "        mesh_file = MemoryFile(json.dumps(gltf_json), \".gltf\", \"sphere.gltf\")\n",
    "        mesh_data = InMemoryMesh(mesh_file=mesh_file,\n",
    "                                 supporting_files=self._gltf_supporting_files)\n",
    "        return Mesh(mesh_data=mesh_data)\n",
    "\n",
    "    def add_ball(self, pbr_material={}, position=[0, 0, 1]):\n",
    "        \"\"\"Add a \"white\" ball so we can observe various lighting effects on it.\n",
    "        If `pbr_material` is None, the ball is a Sphere primitive with Phong\n",
    "        materials. Otherwise, the pbr_material is applied to the sphere.gltf\n",
    "        file.\n",
    "\n",
    "        The ball (sphere with radius = 1) is placed at position, p_WB.\n",
    "        \"\"\"\n",
    "        shape = self._get_sphere_shape(pbr_material)\n",
    "        ball = GeometryInstance(X_PG=RigidTransform(position),\n",
    "                                shape=shape, name=f\"ball{self._ball_count}\")\n",
    "        self._ball_count += 1\n",
    "        # When reading the gltf, we don't want to assign a diffuse property.\n",
    "        self._add_geometry(ball, None)\n",
    "\n",
    "    def _add_geometry(self, geometry, diffuse_color = Rgba(1,1,1)):\n",
    "        self._assign_materials(geometry, diffuse_color)\n",
    "        self._scene_graph.RegisterGeometry(\n",
    "            context=self._sg_context,\n",
    "            source_id=self._source_id,\n",
    "            frame_id=self._scene_graph.world_frame_id(),\n",
    "            geometry=geometry)\n",
    "\n",
    "    def set_render_engines(self, renderer_specs):\n",
    "        \"\"\"Clears all render engines from the diagram, adding the engines\n",
    "        implied by the `renderer_specs` (see RenderEngineSpecs below).\n",
    "        \"\"\"\n",
    "        for camera in self._cameras:\n",
    "            self._scene_graph.RemoveRenderer(self._sg_context,\n",
    "                                             camera.core().renderer_name())\n",
    "        assert self._scene_graph.RendererCount(self._sg_context) == 0\n",
    "        self._cameras.clear()\n",
    "\n",
    "        for renderer in renderer_specs:\n",
    "            # We're only doing PBR, so force PBR to true.\n",
    "            pbr_params = renderer.params | {\"force_to_pbr\": True}\n",
    "            self._scene_graph.AddRenderer(\n",
    "                context=self._sg_context, name=renderer.label,\n",
    "                renderer=MakeRenderEngineVtk(\n",
    "                    RenderEngineVtkParams(**pbr_params)))\n",
    "            self._cameras.append(self._make_camera(renderer.label))\n",
    "\n",
    "    def render(self):\n",
    "        \"\"\"Renders an image for each of the \"\"\"\n",
    "        # Bumping the time dirties the dependency tree; CameraController will\n",
    "        # query for a new camera pose.\n",
    "        self._context.SetTime(self._context.get_time() + 1)\n",
    "        query_object = self._diagram.GetOutputPort(\"query_object\").Eval(\n",
    "            self._context)\n",
    "\n",
    "        MAX_COLS = 2\n",
    "        if len(self._cameras) <= MAX_COLS:\n",
    "           num_cols = len(self._cameras)\n",
    "           num_rows = 1\n",
    "        else:\n",
    "           num_cols = MAX_COLS\n",
    "           num_rows = int(np.ceil(len(self._cameras) / MAX_COLS))\n",
    "        width = 7 * num_cols\n",
    "        height = 5 * num_rows\n",
    "        fig, ax = plt.subplots(num_rows, num_cols, figsize=(width, height),\n",
    "                               squeeze=False)\n",
    "        fig.set_facecolor('white')\n",
    "        ax = ax.reshape(-1, )\n",
    "        fig.subplots_adjust(hspace=0.1, left=0.05, right=0.95)\n",
    "\n",
    "        images = []\n",
    "        for i, camera in enumerate(self._cameras):\n",
    "            image = query_object.RenderColorImage(\n",
    "                camera=camera, parent_frame=self._cam_controller.frame_id(),\n",
    "                X_PC=RigidTransform())\n",
    "            ax[i].imshow(image.data)\n",
    "            ax[i].set_title(camera.core().renderer_name())\n",
    "            ax[i].set_axis_off()\n",
    "        for i in range(len(self._cameras), len(ax)):\n",
    "            ax[i].set_axis_off()\n",
    "        plt.pause(0.5)\n",
    "\n",
    "        # Clear any hard-coded camera pose.\n",
    "        self.set_override_camera_pose(None)\n",
    "\n",
    "\n",
    "class RenderEngineSpec:\n",
    "    def __init__(self, label: str, params: dict):\n",
    "        \"\"\"Specification for a render engine.\n",
    "        `label` will be used as the renderer name and the image label.\n",
    "        `params` is a dictionary that can be used as:\n",
    "            `RenderEngineVtkParams(**spec).\n",
    "        \"\"\"\n",
    "        self.label = label\n",
    "        self.params = params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"first-rendering\"></a>\n",
    "## The first rendering\n",
    "\n",
    "Let's test the infrastructure. We'll see if we can render a simple image.\n",
    "\n",
    "We'll also start a meshcat instance here. It is typical to do this once at the\n",
    "beginning of the notebook, and use the same instance throughout.\n",
    "\n",
    "In order to be able to track the camera's position in the Meshcat session,\n",
    "you must open the _modified_ url (the one that ends with `?tracked_camera=on`)\n",
    "which will print out after executing the next block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "meshcat = StartMeshcat()\n",
    "print(f\"The URL to enable camera tracking: {meshcat.web_url()}/?tracked_camera=on\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once meshcat is up and connected, this block will check to make sure that\n",
    "camera tracking is properly enabled. If it fails but you're sure that meshcat\n",
    "is up and running on the correct URL, simply re-evaluate it again until it\n",
    "reports success."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if meshcat.GetTrackedCameraPose() is None:\n",
    "   logging.error(\n",
    "      f\"Make sure you open the URL: {meshcat.web_url()}/?tracked_camera=on\")\n",
    "else:\n",
    "    print(\"Camera tracking enabled!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up the render framework\n",
    "\n",
    "The block below is the first block which sets up the rendering. In this\n",
    "first rendering, we'll instantiate the comparator, populate it with some\n",
    "geometry, push the geometry to meshcat, configure a `RenderEngine`, and,\n",
    "finally, pose the camera.\n",
    "\n",
    "For subsequent renderings, we may not do all of these (for example, several\n",
    "renderings in a row can use the same geometry)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator = RenderComparator(meshcat)\n",
    "\n",
    "# Add some geometry to the scene.\n",
    "comparator.add_ground()\n",
    "comparator.add_ball()\n",
    "\n",
    "# Make sure we push the geometry to the meshcat browser session.\n",
    "comparator.visualize()\n",
    "\n",
    "# Configure a single, default render engine.\n",
    "default_params = {}\n",
    "comparator.set_render_engines([RenderEngineSpec(\"First Rendering\", default_params)])\n",
    "\n",
    "# The *next* invocation of render will use *this* pose. Subsequent invocations\n",
    "# will pull the pose from meshcat.\n",
    "# Look at the ball and ground from above.\n",
    "comparator.pose_camera_once(camera_in_world=[0, 0, 5],\n",
    "                            target_in_world=[0, 0, 0],\n",
    "                            up_in_world=[0, 1, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Invoke rendering\n",
    "\n",
    "In the meshcat visualizer, you should now see a white sphere on a grey\n",
    "plane (well, large, flat box). Evaluating the next block should _render_ an\n",
    "image of the same scene and display that image below the code block.\n",
    "\n",
    "The first rendering is from the camera pose defined above. Subsequent\n",
    "invocations of `render()` will pull the pose from meshcat visualizer. Each\n",
    "time you render, make note of the output `comparator.pose_camera_once(...)`\n",
    "associated with the image. That is the pose command that you can use to\n",
    "reproduce the image shown.\n",
    "\n",
    "Try changing the view in your meshcat session and re-evaluating the next block.\n",
    "Confirm that the render perspective changes with the visualizer's view. If not,\n",
    "confirm the block above, testing the camera feedback, prints out \"Camera\n",
    "tracking enabled!\"\n",
    "\n",
    "Practice manipulating the view. You'll probably be doing that a fair amount\n",
    "during the course of this tutorial.\n",
    "\n",
    "**ATTENTION**: Expect the rendered image to deviate from what is shown in\n",
    "the meshcat browser. Meshcat introduces an independent light set from those\n",
    "defined in the `RenderEngine`. Meshcat also uses a PBR illumination model,\n",
    "so much of what is described for `RenderEngineVtk` will also apply there\n",
    "although the exact details will differ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The default light\n",
    "\n",
    "If no lights are specified (see below), there is a default light: a single\n",
    "light affixed to the camera. That is why the ball always looks the same,\n",
    "regardless of how the camera moves. The light is always shining straight ahead\n",
    "from the camera. However, the ground plane's level of illumination does\n",
    "change based on the viewing angle."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"direct_lighting\"></a>\n",
    "## Direct Lighting\n",
    "\n",
    "You can think of virtual lights as objects that emit light energy in rays. The\n",
    "direction of the rays plays a significant role in how the objects in your scene\n",
    "get illuminated (see below). Basically, the various light types vary in the\n",
    "definition of those light rays.\n",
    "\n",
    "In this section, we'll focus on the *distribution* of light in the scene from\n",
    "each light type. We won't focus too much on specific reflection effects (that\n",
    "requires an understanding of [materials](#pbr_materials)).\n",
    "\n",
    "Drake supports three kinds of lights: point, spot, and directional.\n",
    "\n",
    "  - __Point light__: This is essentially an idealized light bulb. It has zero radius\n",
    "    so all light rays emanate from a single point, parameterized by\n",
    "    `drake::geometry::render::LightParameter::position`. The rays are emitted in\n",
    "    all directions. Defining the light's `position` determines the emitted light\n",
    "    rays' common origin and directions.\n",
    "  - __Spot light__: This is a point light with an opaque, conical light shade on \n",
    "    it, parameterized by `drake::geometry::render::LightParameter`'s fields:\n",
    "    `position`, `direction`, and `cone_angle`. Like a point light, it emits all\n",
    "    light from a single point (located at `position`). However, there is an\n",
    "    oriented cone associated with the spot light. The cone has its apex at\n",
    "    `position` and it expands in the direction indicated by the `direction` vector. Finally, the `cone_angle` controls how narrow the spot light's beam\n",
    "    is. The angle (in degrees) specifies the _half_ angle of the cone's span, or\n",
    "    the angle between the cone's central axis and the boundary of the cone. No\n",
    "    light is emitted outside of this cone. \n",
    "  - __Directional light__: This can be thought of as a point light that is\n",
    "    infinitely far away, or idealized sunlight; all emitted light rays are\n",
    "    parallel. As such, the directional light has no meaningful value for\n",
    "    `position`, but its `direction` value defines the direction of the emitted\n",
    "    light rays.\n",
    "\n",
    "![](https://drake.mit.edu/doxygen_cxx/light_types.png)\n",
    "**Figure:** The three kinds of explicit lights.\n",
    "\n",
    "### Computing diffuse illumination\n",
    "\n",
    "Light interacts with matter in various ways. While the physics is fascinating\n",
    "and intricate, rendering technology typically focuses on the _effects_ and\n",
    "creates algorithms that approximate many of the observable effects. The names\n",
    "you might see in this regard are such things as: \"diffuse\", \"specular\",\n",
    "\"ambient\", \"reflection\", \"refraction\", \"diffraction\", etc., This tutorial won't\n",
    "necessarily discuss all supported phenomenon. Instead, it will touch (and\n",
    "possibly elaborate) on those lighting effects that are most relevant to exploring\n",
    "the rendering configuration.\n",
    "\n",
    "#### Diffuse lighting\n",
    "\n",
    "When light strikes a surfaces, some portion of the light is reflected off of the\n",
    "surface and some portion is refracted into the surface. The light that refracts\n",
    "into the surface may bounce around several times before bouncing out again.\n",
    "The bouncing is a random process. Even though a light ray may strike a surface\n",
    "from a specific direction, the final refracted light can end up shining in an\n",
    "arbitrary direction. Thus, from a viewer's perspective, the observed refracted\n",
    "light doesn't depend on which direction the light ray came from, but simply\n",
    "how much light struck the surface. Furthermore, the light color and intensity\n",
    "can be changed by this process (as some frequencies of light get filtered out\n",
    "and energy gets absorbed as part of the bouncing around).\n",
    "\n",
    "To fully account for this refracting phenomenon, we need to add up all of the\n",
    "light that can strike that surface point from any direction (this is, in fact,\n",
    "a hemisphere of directions; light rays coming from \"behind\" the surface point\n",
    "do not illuminate it). As we sum up all of the light sources and account for\n",
    "the energy loss and color shifts, we end up with the total refracted light\n",
    "emitted from the point on the surface.\n",
    "\n",
    "This process \"diffuses\" the lighting shining onto the surface, blurring,\n",
    "tinting, and dimming it from the viewer's perspective. (Note: we'll discuss\n",
    "the reflected portion down below when we talk about \"specular\" reflections.)\n",
    "\n",
    "The question of \"how much light\" strikes the surface is affected by the\n",
    "intensity of the light sources and the angle at which the light ray strikes\n",
    "the surface. A light ray striking the surface perpendicularly will have the\n",
    "most energy. As the angle becomes shallower, the amount of light striking the\n",
    "surface will get smaller. This is computed based on the normal of the\n",
    "illuminated surface and the direction of light rays produced by each light\n",
    "type.\n",
    "\n",
    "For a scene with a flat plane, we can see the different diffuse lighting effects\n",
    "that the three lights have. We'll render a flat plane with three different lights:\n",
    "a point light 5 meters above the plane, and a spot light 5 meters above the\n",
    "plane, shining straight down, and a directional light shining directly down on\n",
    "the plane."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, let's look at the distribution of light from each light type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "# Three render engines with three different lights. We express the position and\n",
    "# direction of the lights in the *world* frame; the lights will *not* move with\n",
    "# the camera.\n",
    "\n",
    "def make_config(custom_light_params):\n",
    "    # We'll mix in custom light parameters with these default parameters. Not\n",
    "    # all parameters will have effect on all light types.\n",
    "    light = LightParameter(**({\"frame\": \"world\",\n",
    "                               \"position\": [0, 0, 5],\n",
    "                               \"cone_angle\": 50,\n",
    "                               \"direction\": [0, 0, -1]\n",
    "                              } | custom_light_params))\n",
    "    return {\"lights\": [light]}\n",
    "\n",
    "comparator.set_render_engines([\n",
    "    RenderEngineSpec(\"Point Light\", make_config({\"type\": \"point\"})),\n",
    "    RenderEngineSpec(\"Spot Light\", make_config({\"type\": \"spot\"})),\n",
    "    RenderEngineSpec(\"Directional Light\", make_config({\"type\": \"directional\"})),\n",
    "    ])\n",
    "\n",
    "# Pick a view above the ground, but not straight down.\n",
    "comparator.pose_camera_once(camera_in_world=[0, -1, 10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Don't forget, repeated executions will\n",
    "[render from your meshcat camera perspective](#Invoke-rendering)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some essential differences between the three images. Let's look at\n",
    "those differences and see how light choice effects them.\n",
    "\n",
    "The most obvious difference is the spot light's limit. It doesn't illuminate\n",
    "the full ground box. We see a circular pool of light. The light is circular\n",
    "because the spot light's cone of light intersects the ground perpendicularly.\n",
    "If the spot light were tilted, the pool of light would be elliptical.\n",
    "In `RenderEngineVtk`, the spot light's pool of light *always* has a hard edge.\n",
    "\n",
    "A more subtle difference is between the point light and directional light\n",
    "images. In both images, the ball's appearance is essentially the same. However,\n",
    "the ground box is different. With the point light, the ground nearest the ball\n",
    "is the most brightly illuminated. Away from the ball, the ground is much darker.\n",
    "In contrast, the ground box is the same color everywhere with the directional\n",
    "light. It's also worth pointing out that _within_ the pool of light, the spot\n",
    "light's pattern of illumination matches the point light. Why is the directional\n",
    "light different?\n",
    "\n",
    "In order for a surface to be visible (to eye or camera), light must strike\n",
    "the surface. The more light that strikes it, the brighter the surface appears.\n",
    "Three light properties determine the amount of light striking the surface:\n",
    "direction, intensity and attenuation.\n",
    "\n",
    "#### Direction\n",
    "\n",
    "For a beam of light with a fixed energy level, the portion of that\n",
    "energy that actually reflects off a surface depends on the angle at which\n",
    "the beam strikes the surface. If the light strikes the surface perpendicular\n",
    "to the surface, the energy (per unit area) is maximized. As the beam's\n",
    "approach direction diverges from perpendicular, less energy per unit area\n",
    "strikes the surface, so there's less to reflect. The surface will appear\n",
    "dimmer. As the light direction becomes parallel to the surface, the surface\n",
    "will not be illuminated at all. Obviously, any light beam coming from _behind_\n",
    "an opaque surface will not illuminate it all.\n",
    "\n",
    "So, how does that apply to the images above?\n",
    "\n",
    "The point light and directional light (and spot light, of course) have the\n",
    "same intensity. The point light is placed above the ball so that light\n",
    "striking the top of the ball strikes the ball perpendicularly. Similarly,\n",
    "the directional light is aimed straight down. That is why the ball in both\n",
    "images appears more or less identical.\n",
    "\n",
    "However, as we consider the ground far away from the ball, the relationship\n",
    "between surface orientation and light direction differs. By definition,\n",
    "the rays from the directional light are the same throughout the whole scene.\n",
    "That's why the ground has constant illumination across its visible surface.\n",
    "However, the direction from surface to point light changes as we move away\n",
    "from the center; less light is striking those areas because of the shallower\n",
    "angle.\n",
    "\n",
    "Try changing the direction of the directional light or the position of the\n",
    "point light in the previous rendering to see how illumination levels change."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Intensity\n",
    "\n",
    "Intensity is simple to understand. It is simply how much energy is in the\n",
    "electromagnetic radiation. In\n",
    "[`LightParameter`](https://drake.mit.edu/doxygen_cxx/structdrake_1_1geometry_1_1render_1_1_light_parameter.html),\n",
    "the `intensity` controls this energy level (see below). As the light intensity\n",
    "increases, the amount of energy reaching the camera can exceed the camera's\n",
    "ability to display. Details are lost as the pixels saturate to white.\n",
    "Too little intensity, and the image becomes an indistinguishable black blob."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_config(intensity):\n",
    "    light = LightParameter(**{\"frame\": \"world\",\n",
    "                              \"intensity\": intensity,\n",
    "                              \"type\": \"point\",\n",
    "                              \"position\": [0, 0, 4],\n",
    "                             })\n",
    "    return {\"lights\": [light]}\n",
    "\n",
    "comparator.set_render_engines([\n",
    "    RenderEngineSpec(f\"Intensity = {i}\", make_config(i))\n",
    "    for i in [0.25, 1.0, 15.0]])\n",
    "\n",
    "# Pick a view above the ground, but not straight down.\n",
    "comparator.pose_camera_once(camera_in_world=[0, -1, 10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Attenuation\n",
    "\n",
    "In the real world, light intensity decreases as the distance to the light\n",
    "increases. This attenuation of physical light follows the inverse-squared law --\n",
    "if distance doubles, intensity falls to a quarter. The lights above have\n",
    "_no_ attenuation. The distance to the light has no bearing on light intensity.\n",
    "For truly realistic scene lighting, we should change the lights to have\n",
    "quadratic attenuation.\n",
    "\n",
    "[`LightParameter`](https://drake.mit.edu/doxygen_cxx/structdrake_1_1geometry_1_1render_1_1_light_parameter.html)\n",
    "also includes attenuation parameters. It is a triple of coefficients: (c, l, q).\n",
    "A light's intensity gets attenuated based on the distance, `d`, between\n",
    "light source and illuminated point. The scale factor is:\n",
    "\n",
    "    total_attenuation = 1 / (c + l * d + q * d²)\n",
    "    \n",
    "The inverse-squared law is satisfied by c = 0, l = 0, q = 1. In practice,\n",
    "it sometimes becomes convenient to have the intensity decay more slowly by\n",
    "reintroducing constant or linear factors. The sum of the three coefficients\n",
    "should total one. If the sum is less than one, light intensity will get boosted.\n",
    "If all three coefficients are zero, the light has infinite intensity.\n",
    "\n",
    "**Note**: Attenuation does not apply to directional lights as there is\n",
    "no concept of \"distance\" with those lights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_config(attenuation):\n",
    "    light = LightParameter(**{\"frame\": \"world\",\n",
    "                              \"intensity\": 2,\n",
    "                              \"type\": \"point\",\n",
    "                              \"position\": [0, 0, 4],\n",
    "                              \"attenuation_values\": attenuation,\n",
    "                             })\n",
    "    return {\"lights\": [light]}\n",
    "\n",
    "comparator.set_render_engines([\n",
    "    RenderEngineSpec(\"Constant\", make_config(attenuation=[1, 0, 0])),\n",
    "    RenderEngineSpec(\"Linear\", make_config(attenuation=[0, 1, 0])),\n",
    "    RenderEngineSpec(\"Quadratic\", make_config(attenuation=[0, 0, 1])),\n",
    "    ])\n",
    "\n",
    "# Pick a view above the ground, but not straight down.\n",
    "comparator.pose_camera_once(camera_in_world=[0, -1, 10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combining lights\n",
    "\n",
    "A scene can include multiple lights. Sometimes, it is necessary to use multiple\n",
    "lights to create the final effect of real-world lighting. For `RenderEngineVtk`\n",
    "you can declare as many lights as you like, although more lights mean greater\n",
    "computational cost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "# Add several arbitrary lights; each with an arbitrarily different color to\n",
    "# better distinguish the lights.\n",
    "\n",
    "comparator.set_render_engines([\n",
    "    RenderEngineSpec(\n",
    "        \"Multiple Lights\",\n",
    "        {\"lights\": [LightParameter(type=\"point\",\n",
    "                                   frame=\"world\",\n",
    "                                   color=Rgba(1, 0, 0),\n",
    "                                   position=[-2,-2, 1]),\n",
    "                    LightParameter(type=\"point\",\n",
    "                                   frame=\"world\",\n",
    "                                   color=Rgba(0, 1, 0),\n",
    "                                   position=[-2, 2, 1]),\n",
    "                    LightParameter(type=\"point\",\n",
    "                                   frame=\"world\",\n",
    "                                   color=Rgba(0, 0, 1),\n",
    "                                   position=[2, 2, 1]),\n",
    "                    LightParameter(type=\"point\",\n",
    "                                   frame=\"world\",\n",
    "                                   position=[2, -2, 1]),\n",
    "                    LightParameter(type=\"spot\",\n",
    "                                   frame=\"world\",\n",
    "                                   color=Rgba(0, 0.8, 0.8),\n",
    "                                   # Note: the direction vector doesn't have to\n",
    "                                   # be unit length.\n",
    "                                   direction=np.array((-1, 0, -1)),\n",
    "                                   position=[3, 0, 3],\n",
    "                                   cone_angle=35)\n",
    "                                   ]}),\n",
    "    ])\n",
    "\n",
    "# Pick a view above the ground, but not straight down.\n",
    "comparator.pose_camera_once(camera_in_world=[0, -1, 6])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"shadows\"></a>\n",
    "## Shadows\n",
    "\n",
    "None of the previous renderings had any shadows. Everything was illuminated as\n",
    "if there were nothing between the surface and the light. In the real world,\n",
    "light can be _occluded_. Physical objects block the light, casting shadows onto\n",
    "surfaces that would otherwise have been lit. In rendering, we don't get\n",
    "occlusion for free. We create shadows through a variety of approximations. The\n",
    "spot light's cone of light is our first example. In the real world, that cone of\n",
    "light is created by a physical shade that blocks emitted light. The virtual\n",
    "spot light is simply modeled as a cone of light directly. However, this\n",
    "simplified cone model, while sufficient for creating a _pool_ of light, is\n",
    "insufficient for shadows.\n",
    "\n",
    "There are multiple techniques to detect occlusion. `RenderEngineVtk` uses a\n",
    "technique called \"shadow maps\". Shadow maps are an encoding of surfaces that are\n",
    "visible *to the light source*. Any surface that is visible gets illuminated.\n",
    "Surfaces that are not visible are in shadow. A shadow map is a texture map\n",
    "(hence the name), where every pixel contains a value storing the distance of\n",
    "the closest surface in the direction of the pixel. These distance measurements\n",
    "are reconciled at rendering time to determine if a surface is in shadow or not.\n",
    "\n",
    "This shadow model is binary. A pixel is either visible from the light or not\n",
    "(up to possible numerical precision issues). Given that our lights are all\n",
    "modeled as point sources (more or less), this binary illumination classification means that all shadows will have hard edges; there is no blurry\n",
    "[penumbra](https://en.wikipedia.org/wiki/Umbra,_penumbra_and_antumbra).\n",
    "\n",
    "Let's explore these phenomena. But first, let's make sure shadows work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "# One renderer casts shadows, one doesn't. Otherwise identical.\n",
    "\n",
    "def make_config(custom_config):\n",
    "    config = {\"lights\": [LightParameter(type=\"spot\",\n",
    "                                        frame=\"world\",\n",
    "                                        position=[0, 0, 5],\n",
    "                                        cone_angle=50,\n",
    "                                        direction=[0, 0, -1])]}\n",
    "    return config | custom_config\n",
    "\n",
    "comparator.set_render_engines([\n",
    "    RenderEngineSpec(\"With Shadows\", make_config({\"cast_shadows\": True})),\n",
    "    RenderEngineSpec(\"No Shadows\", make_config({\"cast_shadows\": False}))])\n",
    "\n",
    "comparator.pose_camera_once(camera_in_world=[0, -3, 3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should clearly see that the ball casts a shadow on the ground plane in the\n",
    "image on the left. The image on the right is a repeat from before, but has\n",
    "*explicitly* disabled shadow casting. If you see the shadow, you can proceed\n",
    "with the next steps to see how the various settings control the shadow\n",
    "appearance.\n",
    "\n",
    "(Keen observers will no doubt notice the pixely edge of the shadow. That topic\n",
    "is explored below in the \"Shadow resolution\" section.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What lights can cast shadows?\n",
    "\n",
    "Spot lights and directional lights can cast shadows. Point lights cannot. This\n",
    "is not a fundamental property of the model, but one of how `RenderEngineVtk` is\n",
    "implemented. We'll get a bit more into the implementation details and how they\n",
    "relate to this when we consider the limitations on spot light shadows below.\n",
    "\n",
    "The following renderings will enable shadows and each uses a unique light type.\n",
    "Notice that we've changed the light position (bringing the spot and point\n",
    "lights closer to the ball)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "# The properties common to all declared lights (some values may\n",
    "# be ignored -- e.g., cone_angle for point lights).\n",
    "# cast_shadows is enabled for all.\n",
    "def make_config(custom_light_params):\n",
    "   light = LightParameter(**({\"frame\": \"world\",\n",
    "                              \"position\": [0, 0, 3],\n",
    "                              \"direction\": [0, 0, -1],\n",
    "                              \"cone_angle\": 50} |\n",
    "                             custom_light_params))\n",
    "   return {\"lights\": [light], \"cast_shadows\": True}\n",
    "\n",
    "comparator.set_render_engines([\n",
    "    RenderEngineSpec(f\"{light.capitalize()} Light\",\n",
    "                     make_config({\"type\": light}))\n",
    "    for light in [\"point\", \"spot\", \"directional\"]])\n",
    "\n",
    "# Pick a view above the ground, but not straight down.\n",
    "comparator.pose_camera_once(camera_in_world=[0, -2.5, 3.5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As promised, the point light casts no shadow on the ground plane. Declaring\n",
    "`cast_shadows = True` is not an error for point lights; the value simply has no\n",
    "effect. Conversely, the spot and directional lights both cast shadows, there are\n",
    "some obvious differences between the shadows.\n",
    "\n",
    "- The shadow cast from the directional light is smaller than that of the spot\n",
    "  light. Not only that, it even appears smaller than the ball.\n",
    "   - The directional shadow *is* definitely smaller than the spot light shadow.\n",
    "     Because the spot light's light rays diverge, the shadow will be larger than\n",
    "     the object casting it. This is illustrated in the figure below in the\n",
    "     shadow map resolution section.\n",
    "   - However, the directional shadow is *not* smaller than the ball itself. This\n",
    "     is an optical illusion due to perspective. The ball's shadow is, in fact,\n",
    "     the same size as the ball itself. But the shadow is farther from the camera\n",
    "     than the ball itself. And in perspective, the closer something is, the\n",
    "     larger it appears. Try moving the camera and re-rendering.\n",
    "     When the camera is close to the ground, it is clear that the shadow has\n",
    "     the same radius as the ball.\n",
    "- The spot light's shadow has a much smoother edge than the directional light.\n",
    "  The directional light's shadow has a clearly pixelated edge. This is due to\n",
    "  shadow map resolution (see below).\n",
    "\n",
    "As a side note, the lighting levels on the ball from the directional light\n",
    "is now quite different from that of the point and spot lights. We moved the\n",
    "point and spot lights *closer* to the ball. So, the surface normal direction\n",
    "and direction to light deviates more quickly along the surface of the ball,\n",
    "changing the effect of the lighting. If we were to move the spot and point\n",
    "lights *farther* from the sphere, the effect would quickly become\n",
    "indistinguishable from the directional light."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Spot light limitations\n",
    "\n",
    "While spot lights can cast shadows, their ability is limited by the\n",
    "cone half-angle (the `cone_angle` parameter). It can cast shadows with a \n",
    "`cone_angle` value *up to* 90 degrees. The following renderings show\n",
    "how a too-large `cone_angle` eliminates shadows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "# Place a single spot light above the ball and vary the cone angles.\n",
    "\n",
    "def make_config(custom_light_params):\n",
    "    light = LightParameter(**{\"type\": \"spot\",\n",
    "                              \"frame\": \"world\",\n",
    "                              \"position\": [0, 0, 6],\n",
    "                              \"direction\": [0, 0, -1]}\n",
    "                             | custom_light_params)\n",
    "    return {\"lights\": [light], \"cast_shadows\": True}\n",
    "\n",
    "comparator.set_render_engines([\n",
    "    RenderEngineSpec(f\"{cone_angle}-degree Spot Light\",\n",
    "                     make_config({\"cone_angle\": cone_angle}))\n",
    "    for cone_angle in [50, 85, 90]])\n",
    "\n",
    "# Pick a view above the ground, but not straight down.\n",
    "comparator.pose_camera_once(camera_in_world=[0, -2.5, 3.5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get a ball shadow for the 50- and 85-degree spot lights. The 90-degree spot\n",
    "light casts no shadow.\n",
    "\n",
    "The reason for this is related to `RenderEngineVtk`'s implementation. It is the\n",
    "same reason that point lights cast no shadows. The entire space visible to the\n",
    "light is mapped to a single 2D image via a simple projection. If every possible\n",
    "shadow a light can cast projects onto a single plane, then that light can\n",
    "cast shadows. This is true for a spot light with `cone_angle < 90 degrees`. The\n",
    "plane whose surface normal is parallel with the light's directional vector can\n",
    "catch all of the shadows. But once the `cone_angle` value reaches 90-degrees,\n",
    "there are emitted light rays that are *parallel* to that plane. They will\n",
    "never intersect with the plane and therefore can't cast shadows onto it. In\n",
    "`RenderEngineVtk`, shadows are simply disabled for such spot lights.\n",
    "\n",
    "The second thing to note about the images is that the _quality_ of the shadows.\n",
    "The shadows from the 50-degree spot light and the 85-degree spot light are\n",
    "obviously different. The shadow from the 85-degree spot light has been reduced\n",
    "to a single, slightly blurry rectangle. This is a good time to get into shadow\n",
    "map resolution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Shadow map resolution\n",
    "\n",
    "The shadow map is a rendering of the scene from the camera's perspective. Each\n",
    "pixel stores the distance from camera to the nearest surface in the direction of\n",
    "the pixel. This makes it closely resemble a depth map. The following\n",
    "illustration shows a simple scene with a sphere and box being illuminated by a\n",
    "spot light.\n",
    "\n",
    "![](https://drake.mit.edu/doxygen_cxx/shadow_map_resolution.png)\n",
    "**Figure:** Shadow map resolution: the resolution of the shadow map will\n",
    "determine how much detail is present in the shadow. Too little resolution,\n",
    "and the blocky shadow map pixels become apparent in the final rendering.\n",
    "\n",
    "Below the scene are visualizations of possible shadow maps, each with different\n",
    "resolution. The \"Full depth map\" represents the idealized shadow map. In this\n",
    "case, we've colored \"far\" values as black and \"near\" values as white. Note that\n",
    "although the sphere is smaller than the box, because it is closer to the spot\n",
    "light, it has a larger footprint in the shadow map. This idealized map doesn't\n",
    "have pixels, *per se*.\n",
    "\n",
    "The next row down shows a coarse approximation of the idealized map. In this\n",
    "case, the domain of the shadow map has been reduced to six pixels. The coarse\n",
    "sampling leads to significant aliasing issues. The extents of the sphere and\n",
    "box in the shadow map have changed significantly. It is easy to imagine that\n",
    "for a 3D scene, the shadow of the sphere will be turned into a single square\n",
    "pixel, just as we observed above from the 85-degree spotlight.\n",
    "\n",
    "The final row doubles the resolution over the previous row. There are still\n",
    "sampling artifacts, but the approximation of the actual shadow boundaries has\n",
    "definitely improved.\n",
    "\n",
    "Warning: There is a cost for each shadow-casting light. The scene must be\n",
    "rendered once for each light _at each frame_ to create that light's up-to-date\n",
    "shadow map. The more lights that cast shadows, the more times the scene has to\n",
    "be rendered to produce one output image.\n",
    "\n",
    "Let's look at the `RenderEngineVtkParams::shadow_map_size` parameter and see\n",
    "its impact on the shadows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "# Compare different levels of shadow resolution.\n",
    "\n",
    "def make_config(custom_config):\n",
    "    light = LightParameter(type=\"spot\", frame=\"world\",\n",
    "                           position=[0, 0, 4], cone_angle=70,\n",
    "                           direction=[0, 0, -1])\n",
    "    return {\"lights\": [light], \"cast_shadows\": True} | custom_config\n",
    "\n",
    "comparator.set_render_engines([\n",
    "    RenderEngineSpec(f\"{resolution} pixel map - spot\",\n",
    "                     make_config({\"shadow_map_size\": resolution}))\n",
    "    for resolution in [128, 256, 512, 1024]])\n",
    "\n",
    "# Pick a view above the ground, but not straight down.\n",
    "comparator.pose_camera_once(camera_in_world=[0, -2.5, 3.5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a fixed light, we can easily change the shadow map resolution to improve the\n",
    "visual fidelity of the shadow edges. But this is a setting that needs to be used\n",
    "judiciously. It would be \"safe\" to simply set all shadow maps to some large\n",
    "(power-of-two) value to make sure that shadows all look clean. However, this\n",
    "comes at a cost. Each shadow map must be re-rendered with each frame. The speed\n",
    "of this depends on the rendering capability of your computer. It might be\n",
    "reasonable to *try* setting shadow map resolution high straight across the board\n",
    "and see if the performance is appreciably harmed. But, generally, the best\n",
    "strategy is to use the lowest resolution map that provides a visually acceptable\n",
    "shadow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Shadow map registration\n",
    "\n",
    "It is one thing to define the resolution of the image that stores the shadow\n",
    "map. But what is the relationship of that image and the rendered world? In\n",
    "other words, how big are those pixels? The bigger the pixels, the more likely\n",
    "you are to notice the resolution.\n",
    "\n",
    "The shadow map has to be \"registered\" to the world in some way -- defining which\n",
    "portion of the world is accounted for by the shadow map. The image is\n",
    "finite, the world is, in principle, infinite. So, only some *portion* of the\n",
    "world can be included in the domain of the shadow map. There are no parameters\n",
    "for controlling this domain of shadow coverage. So, how is it defined?\n",
    "\n",
    "Both shadow-casting lights have their domains automatically and dynamically\n",
    "computed on every rendering. At their core, each light determines the extent\n",
    "of those objects that *can* be illuminated by the light, and registers the\n",
    "shadow map to cover that full domain.\n",
    "\n",
    "This domain is updated on a render-by-render basis to prevent objects moving in\n",
    "and out of regimes in which shadows can be cast. However, it means if the extent\n",
    "of the scene changes significantly from render to render, the shadows may\n",
    "qualitatively change as well (e.g., previously smooth shadows become blocky).\n",
    "\n",
    "Each shadow-casting light uses its own proprietary logic for determining the\n",
    "illuminated domain.\n",
    "\n",
    "##### Spot light registration\n",
    "\n",
    "![spotlight-shadow-registration](https://drake.mit.edu/doxygen_cxx/shadow_map_registration_spot.png)\n",
    "\n",
    "The spot light determines the domain of its shadow map (its light volume)\n",
    "by creating a view frustum. The frustum is oriented so that the near and\n",
    "far planes are perpendicular to the light's `direction` vector. The planes\n",
    "are scaled so that the cover the extent of the cone; the frustum is\n",
    "tapered towards the source. The near plane is directly in front of the light\n",
    "source itself. The far plane is moved away from the light's `position` in\n",
    "the light `direction` until it as far as the *farthest* scene element *in\n",
    "that direction*.\n",
    "\n",
    "**Implications**\n",
    "\n",
    "1. No matter how \"deep\" the scene is in the direction of the camera's view,\n",
    "   the shadow pixels at any depth will be the same size as the size of\n",
    "   the pixels largely only depends on the *cone angle* of the light. So,\n",
    "   simply fixing cone angle and then modifying the map resolution size\n",
    "   will define the pixelation on the shadow boundaries.\n",
    "2. If the scene is \"deep\" in the direction of the camera's view, but we're\n",
    "   only rendering the items nearest the camera, we will effectively have\n",
    "   reduced precision in the depth calculations. This only affects thin\n",
    "   objects piled on top of each other.\n",
    "\n",
    "##### Directional light registration\n",
    "![directional-shadow-registration](https://drake.mit.edu/doxygen_cxx/shadow_map_registration_directional.png)\n",
    "\n",
    "Directional lights don't have finite extent. They can, in principle, cast light\n",
    "anywhere in the world. In this case, the extent of the illuminated domain is\n",
    "defined by the extent of the geometry that *can* be illuminated. Essentially, we\n",
    "create a bounding *cube* around all of the geometry aligned to the light's\n",
    "direction vector. The size of the cube face is the size of the shadow map in the\n",
    "world.\n",
    "\n",
    "**Implications**\n",
    "\n",
    "1. The more the scene spreads out perpendicular to the light's direction, the\n",
    "   more shadow map resolution you need. Rendering scenes with ground planes\n",
    "   can cause the shadow volume to increase to enclose a volume much larger\n",
    "   than the subject of the camera. This will require much higher shadow\n",
    "   map resolution to get a good effect near the subject, wasting much of the\n",
    "   shadow map resolution.\n",
    "2. If the shadow map gets \"stretched\" out because of the scene bound, it may\n",
    "   be that enabling shadows will produce no discernible effect. It may simply be\n",
    "   that the shadow map has been stretched so large, that a shadow map pixel is\n",
    "   larger than your target objects; they simply don't render into the shadow map.\n",
    "\n",
    "Let's see what that looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "# Compare different levels of shadow resolution.\n",
    "\n",
    "def make_config(custom_config):\n",
    "    light = LightParameter(type=\"directional\", frame=\"world\",\n",
    "                           position=[0, 0, 4], cone_angle=70,\n",
    "                           direction=[0, 0, -1])\n",
    "    return {\"lights\": [light], \"cast_shadows\": True} | custom_config\n",
    "\n",
    "comparator.set_render_engines([\n",
    "    RenderEngineSpec(f\"{resolution} pixel map - directional\",\n",
    "                     make_config({\"shadow_map_size\": resolution}))\n",
    "    for resolution in [64, 256, 1204]])\n",
    "\n",
    "# Pick a view above the ground, but not straight down.\n",
    "comparator.pose_camera_once(camera_in_world=[0, -1.5, 2.5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given that the light is emitting parallel light rays perpendicularly into the\n",
    "ground, we would expect the ball's shadow to be the same size as the ball itself.\n",
    "It is. Given the focal length of our render camera, we're getting strong\n",
    "perspective effects -- things farther from the camera are scaled smaller than\n",
    "things closer. When the ball is closer to the camera than its shadow, it will\n",
    "appear larger. When the shadow is closer to the camera than the ball, the\n",
    "shadow will appear bigger. The important thing to note is that these shadows are\n",
    "both expected and correct."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"pbr_materials\"></a>\n",
    "### PBR Materials\n",
    "\n",
    "The previous section was all about controlling a source of light (not\n",
    "the _only_ source of light, more on that [later](#environment_maps)).\n",
    "However, emitting light is only the start. We're interested in how it\n",
    "_reflects_ off of surfaces to the camera. To understand that, we need\n",
    "to talk about how materials are modeled in the PBR model. If you have\n",
    "the wrong model in your mind, you might be surprised why changing\n",
    "light properties doesn't have the effect you expect.\n",
    "\n",
    "Again, this discussion is going to be heavily biased towards intuition and\n",
    "principle. Visit one of the links in the introduction to get into the\n",
    "fine details of the material definitions.\n",
    "\n",
    "#### Microfacts and reflections\n",
    "\n",
    "We see **specular** reflections every day. These are the mirror-like\n",
    "reflections such that images of the environment are visible on the\n",
    "surface. Even some of the most non-mirror-like surfaces exhibit some\n",
    "degree of specular reflection -- dull concrete surfaces will still show\n",
    "highlights when you look at it from the right angle. Take a look at <a href=\"http://filmicworlds.com/blog/everything-is-shiny/\">\n",
    "this site</a> to see how even the \"dullest\" objects have specular\n",
    "reflections.)\n",
    "\n",
    "The physics of these reflections is conceptually simple. Light strikes a\n",
    "point on a surface (with an associated surface normal). The incoming\n",
    "light ray is reflected across the normal. If your eye is on the path of\n",
    "the outgoing ray, you see the reflected light. Another way of saying the\n",
    "same thing is to imagine the \"half-angle vector\" between the incoming\n",
    "light direction and the outgoing view direction. If that vector is\n",
    "aligned with the surface normal, you see a reflection.\n",
    "\n",
    "![simple reflections](https://drake.mit.edu/doxygen_cxx/simple_reflection.png)\n",
    "**Figure:** The light ray strikes the surface with normal _n_ from\n",
    "direction _l_. The three views _v₀_, _v₁_, and _v₂_ form half-angle\n",
    "vectors _h₀_, _h₁_, and _h₂_, respectively. Only _h₁_ aligns with\n",
    "_n_, so only view _v₁_ receives the specular reflection from the\n",
    "light source.\n",
    "\n",
    "In the real world, surface reflectivity can vary widely. Some crisp\n",
    "and clear and others so subtle that you have to look carefully to\n",
    "notice it. These differences come down to the structure of the object\n",
    "surface at a microscopic level.\n",
    "\n",
    "No surface is perfectly smooth. At the microscopic level, irregularties\n",
    "in the material produce many different normals within a small region.\n",
    "As light strikes that region, light will get distributed in many\n",
    "directions simultaneously, based on the distribution of face normal\n",
    "directions. Shiny surfaces are more regular in structure. The majority\n",
    "of microfacets face the same way. Dull surfaces have more chaotic\n",
    "structure. The more coherency in microfacet direction there exists,\n",
    "the shinier the surface and the more distinct the reflection.\n",
    "\n",
    "The effect of this is twofold. First, perceived specular\n",
    "reflections from a highly irregular surface are both dimmer and\n",
    "blurrier. Dimmer, because the light from a single direction gets\n",
    "scattered in multiple directions simultaneously; only a small portion\n",
    "of the incoming light energy from a single direction gets\n",
    "reflected in the viewer's direction. Blurrier, because the light that\n",
    "*does* get reflected in the viewer's direction can come from a\n",
    "larger span of angles.\n",
    "\n",
    "PBR takes a statistical approach to accounting for these microfacets.\n",
    "A single rendered pixel may encompass millions of microfacets.\n",
    "So, in PBR, we describe a surface's \"roughness\". At any given point\n",
    "on the surface, there is an attendant roughness value. From this\n",
    "roughness value, we can determine what portion of the microfacets\n",
    "are aligned with the half-angle vector, _h_. Let's see what that looks\n",
    "like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "comparator = RenderComparator(meshcat)\n",
    "\n",
    "# Add some geometry to the scene.\n",
    "comparator.add_ground()\n",
    "\n",
    "# Increasing roughness, from left to right.\n",
    "test_roughness = [0.05, 0.2, 0.4, 0.6, 0.8, 1.0]\n",
    "dx = 2.2\n",
    "start_x = -(len(test_roughness) - 1) * dx / 2\n",
    "for i, roughness in enumerate(test_roughness):\n",
    "    comparator.add_ball(pbr_material={\"roughnessFactor\": roughness},\n",
    "                        position=[start_x + i * dx, 0, 1])\n",
    "\n",
    "# Make sure we push the geometry to the meshcat browser session.\n",
    "comparator.visualize()\n",
    "\n",
    "comparator.set_render_engines([\n",
    "    RenderEngineSpec(\"Varying roughness\", {}),\n",
    "])\n",
    "\n",
    "# Center on the ball, with it largely filling the frame.\n",
    "comparator.pose_camera_once(camera_in_world=[0, -5.5, 5.5],\n",
    "                            target_in_world=[0, 0, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As roughness increases, the specular reflection of the light grows\n",
    "larger (i.e. blurrier) and dimmer. This is consistent with PBR's\n",
    "goal to conserve energy.\n",
    "\n",
    "##### Clarifications on specular reflections\n",
    "\n",
    "1. The light sources described above are all, in some sense,\n",
    "point lights. As the surface gets smoother (roughness going to zero),\n",
    "the reflection becomes a more exact reflection. Mathematically, when\n",
    "roughness is zero, the specular reflection of light sources are\n",
    "mathematical points. These points would *not* appear in the rendering;\n",
    "the measure of the point is infinitely small. This is correct, with\n",
    "respect to the PBR model, but counterintuitive when modeling the\n",
    "physical world. When computing the specular reflection from *direct*\n",
    "light sources, Drake makes sure the specular highlight does not\n",
    "completely disappear.\n",
    "2. The lighting model does *not* include specular reflections of one geometry\n",
    "in the surface of another. Getting mirrored surfaces that reflect\n",
    "other geometry requires a different kind of technique -- one that\n",
    "`RenderEngineVtk` does not support yet. So, the only specular\n",
    "reflections we will see are those from scene artifacts that are\n",
    "considered to be light sources, e.g., the [explicit lights](#direct_lighting)\n",
    "described above and the [environment maps](#environment_maps) described\n",
    "below.\n",
    "3. Specular reflections always depend on three things: the direction the\n",
    "light ray comes from when striking the surface, the orientation of the\n",
    "surface (its normal direction), _and_ the direction from the surface to\n",
    "the camera.\n",
    "4. When light strikes a surface at a \"grazing\" angle, the surface exhibits\n",
    "greater reflectivity (called\n",
    "[Fresnel reflections](https://en.wikipedia.org/wiki/Fresnel_equations))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Diffuse illumination and \"metallness\"\n",
    "\n",
    "Specular reflections come from the light that bounces off a surface.\n",
    "However, not all light reflects off. Some light refracts into\n",
    "the surface. Depending on the material, some of the light\n",
    "bounces around inside a bit, getting filtered by the material,\n",
    "and bounces out again. This is what makes the surface visible\n",
    "away from areas of perfect specular reflections.\n",
    "\n",
    "In contrast to specular reflections, the diffuse illumination\n",
    "does _not_ depend on the direction from surface to viewer. Diffuse\n",
    "illumination only depends on the available light sources and the\n",
    "surface's normal.\n",
    "\n",
    "PBR models two kinds of materials: metallic and dielectric.\n",
    "Metallic materials are, as the name implies, made of metal.\n",
    "Dielectric materials are, more or less, all other non-metallic\n",
    "materials (e.g., plastic, concrete, wood, glass, etc.).\n",
    "\n",
    "These two materials behave very differently with respect to\n",
    "the light.\n",
    "\n",
    "  - **metallic**\n",
    "    - Metallic materials absorb _all_ refracted light. That means\n",
    "      that they don't exhibit _any_ diffuse reflection. Away from\n",
    "      specular reflections, they are essentially not illuminated.\n",
    "    - Specular reflections get heavily tinted by the metal's\n",
    "      intrinsic color (its \"albedo\"). If you look at gold, copper,\n",
    "      etc., you'll see the reflections in those materials have\n",
    "      been tinted to the metal's underlying color.\n",
    "  - **dielectric**\n",
    "    - Dielectric materials exhibit diffuse reflection. Light does\n",
    "      escape from the surface away from the specular reflections.\n",
    "    - Specular reflections tend to be dimmer than with metallic\n",
    "      materials (the ratio of how much light gets reflected vs.\n",
    "      refracted is much lower). Furthermore, the reflections are\n",
    "      not tinted by the material's albedo.\n",
    "      \n",
    "Although, both metallic and dielectric surfaces can display Fresnel\n",
    "reflections, the effect is typically more noticable on dielectric\n",
    "materials, because their _base_ reflectivity is generally much lower\n",
    "than that of metals.\n",
    "\n",
    "In PBR, these two properties are modeled using a \"metallic\" property\n",
    "(\"metallicFactor\" in glTF files). In principle, this should be a\n",
    "binary value; a surface is *either* metallic *or* dielectric.\n",
    "However, due to interpolation in texture look up, the model handles\n",
    "all values between zero and one, even if it's not physically\n",
    "meaningful to say a material is halfway between being metallic and\n",
    "dielectric.\n",
    "\n",
    "Let's look at these material differences in play."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator = RenderComparator(meshcat)\n",
    "\n",
    "# Add some geometry to the scene.\n",
    "comparator.add_ground()\n",
    "\n",
    "# Two gold-ish smooth-ish spheres; one metallic (metallicFactor = 1)\n",
    "# and one dielectric (metallicFactor = 0).\n",
    "shiny_gold = {\n",
    "    \"baseColorFactor\": [1, .71, .29, 1.0],  # gold.\n",
    "    \"roughnessFactor\": 0.2                  # quite smooth. \n",
    "}\n",
    "comparator.add_ball(pbr_material=(shiny_gold | {\"metallicFactor\": 1.0}),\n",
    "                    position=[-1.1, 0, 1])\n",
    "comparator.add_ball(pbr_material=(shiny_gold | {\"metallicFactor\": 0.0}),\n",
    "                    position=[1.1, 0, 1])\n",
    "\n",
    "# Make sure we push the geometry to the meshcat browser session.\n",
    "comparator.visualize()\n",
    "\n",
    "comparator.set_render_engines([\n",
    "    RenderEngineSpec(\"Metallic vs Dielectric\", {}),\n",
    "])\n",
    "\n",
    "# Center on the ball, with it largely filling the frame.\n",
    "comparator.pose_camera_once(camera_in_world=[0, -2.25, 2.25],\n",
    "                            target_in_world=[0, 0, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The two spheres are strikingly different. They both have the\n",
    "same albedo (underyling color) and roughness values. All of\n",
    "the differences are attributable to the difference between\n",
    "metallic and dielectric.\n",
    "\n",
    "Obviously, the metallic sphere is quite dark, almost like it\n",
    "isn't being illuminated at all. This is the result of two things\n",
    "we've discussed. Metallic surfaces only exhibit specular\n",
    "reflections (never diffuse), and the only scene artifacts that\n",
    "participate in the specular reflections are light sources; we\n",
    "won't see the dielectric sphere reflected in the metallic\n",
    "sphere.\n",
    "\n",
    "Second, the specular reflection of the light itself differs\n",
    "across the two spheres. The metallic sphere's \"hot spot\" is\n",
    "larger and somewhat oversaturated in the center (we'll deal\n",
    "with the oversaturation when we discuss [exposure](#exposure)\n",
    "later). Again, this is because the metallic surface reflects\n",
    "a larger portion of the incoming light than the dielectric\n",
    "surface, so more radiant energy reaches the eye.\n",
    "\n",
    "Finally, the highlight on the metallic surface has a gold\n",
    "tint to it, one that the dielectric's highlight is missing.\n",
    "This is the metallic surface refracting all of the non-gold\n",
    "light frequencies internally.\n",
    "\n",
    "Try varying the roughness to see how each material type\n",
    "changes. Things get particularly interesting when roughness\n",
    "goes to one.\n",
    "\n",
    "We'll explore some more differences again in the next section:\n",
    "[environment maps](#environment_maps)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"environment_maps\"></a>\n",
    "## Environment Maps\n",
    "\n",
    "The name \"environment map\" probably invites an intuitive interpretation: a\n",
    "painted backdrop. Rather than modeling an entire environment, we simply put an\n",
    "image in the background which we can see directly and reflected in\n",
    "mirror-like surfaces. While that's true, it's not the whole story. An\n",
    "environment map is a light source. That means it *will* participate in\n",
    "specular reflections. But it will also participate in diffuse reflections\n",
    "as well. Essentially, each pixel in the environment map is a light source\n",
    "(more on this later).\n",
    "\n",
    "We'll start with a simple environment map and a mirror ball.\n",
    "\n",
    "The image below shows a ball in a simple environment. The mirror ball sits on\n",
    "a gray plane. Around it are three rectangular area lights. Blue and red lights\n",
    "on either side and a *brighter* white light above it.\n",
    "\n",
    "![environment_map_scene](https://drake.mit.edu/doxygen_cxx/env_color_room_in_situ.png)\n",
    "**Figure:** The physical environment.\n",
    "\n",
    "We can make an environment map by replacing the mirror ball with a panoramic\n",
    "camera. The panoramic image defines what is visible around a point in all\n",
    "directions. Essentially, we're painting on the inside of an infinitely\n",
    "large sphere. But we do so with a rectangular image. The mapping between\n",
    "the rectangular image and the sphere is an [equirectangular projection](https://en.wikipedia.org/wiki/Equirectangular_projection). The height of\n",
    "a pixel spans a fixed solid angle of the sphere. The width of a pixel\n",
    "spans the larget solid angle at the \"equator\" of the image (central,\n",
    "horizontal line). But the width of the pixels get smaller as you near\n",
    "the top and bottom of the image (or north and south poles of the sphere).\n",
    "We'll make use of this when we attempt to predict the lighting contribution\n",
    "of the environment map.\n",
    "\n",
    "![](resources/env_color_room.png)\n",
    "**Figure:** The environment map drawn from the physical scene shown above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "package_map = PackageMap()\n",
    "png_path = package_map.ResolveUrl(\n",
    "    \"package://drake_models/tutorials/environment_maps/env_color_room.png\")\n",
    "\n",
    "plt.imshow(ImageIo().Load(png_path).data)\n",
    "\n",
    "# Also send the environment map to the meshcat session.\n",
    "meshcat.SetEnvironmentMap(png_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The red and blue lights are easily identified in the image. The top\n",
    "of the image is the single white area light above the camera. In a\n",
    "Mercator projection, the north pole gets spread across the full width\n",
    "of the top of the image. The light directly above the camera has\n",
    "received the same treatment.\n",
    "\n",
    "The environment should also have appeared in your meshcat session. Rotate the\n",
    "camera to see the various light panels. (The image is quite low resolution, so\n",
    "the light panels should appear as blurry rectangles.) Ultimately, the full\n",
    "effect of the environment map will be easier to understand when applied to a\n",
    "scene.\n",
    "\n",
    "For the next few scenarios, we're going to disable direct light sources\n",
    "entirely and only look at the effect of the environment map. If you haven't\n",
    "done so already, you should turn off Meshcat's lights by opening the \"Controls\"\n",
    "window an unchecking the blue box next to \"Lights\". This will bring the\n",
    "Meshcat window's version closer in line with the renderings in this\n",
    "tutorial.\n",
    "\n",
    "We'll use the previous set of gold-ish balls and see the effect the\n",
    "environment map has on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "# Add a simple environment map.\n",
    "comparator.set_render_engines([\n",
    "    RenderEngineSpec(\"Mirror Ball\",\n",
    "                     {\"environment_map\": EnvironmentMap(\n",
    "                        skybox=True,\n",
    "                        texture=EquirectangularMap(path=png_path)),\n",
    "                      # Disable the light; only environment map light in the scene.\n",
    "                      \"lights\": [LightParameter(intensity=0)]})])\n",
    "\n",
    "# Center on the ball, with it largely filling the frame.\n",
    "comparator.pose_camera_once(camera_in_world=[0, -2.25, 2.25],\n",
    "                            target_in_world=[0, 0, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The metallic sphere is no longer black with a hot spot (as it was\n",
    "with the single direct point light). This is because the environment\n",
    "map provides light in every direction, giving the metal surface\n",
    "something to reflect. Again, all of the reflections have been\n",
    "tinted the metal's gold-ish albedo color.\n",
    "\n",
    "Similarly, the dielectric sphere is reflecting the environment.\n",
    "The specular reflections are dimmer and untinted as before. But\n",
    "there are some other things going on as well. Let's take a closer\n",
    "look at how the environment map interacts with dieclectric\n",
    "materials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator = RenderComparator(meshcat)\n",
    "\n",
    "# Add some geometry to the scene.\n",
    "comparator.add_ground()\n",
    "\n",
    "# This time, we want two white-ish dielectric spheres,\n",
    "# one smooth, one rough.\n",
    "comparator.add_ball(pbr_material={\"roughnessFactor\": 0.1,\n",
    "                                  \"metallicFactor\": 0},\n",
    "                    position=[0, -1.1, 1])\n",
    "comparator.add_ball(pbr_material={\"roughnessFactor\": 0.9,\n",
    "                                  \"metallicFactor\": 0},\n",
    "                    position=[0, 1.1, 1])\n",
    "\n",
    "# Make sure we push the geometry to the meshcat browser session.\n",
    "comparator.visualize()\n",
    "\n",
    "# Add a simple environment map.\n",
    "comparator.set_render_engines([\n",
    "    RenderEngineSpec(\"Mirror Ball\",\n",
    "                     {\"environment_map\": EnvironmentMap(\n",
    "                        skybox=True,\n",
    "                        texture=EquirectangularMap(path=png_path)),\n",
    "                      # Disable the light; only environment map light in the scene.\n",
    "                      \"lights\": [LightParameter(intensity=0)]})])\n",
    "\n",
    "# Center on the ball, with it largely filling the frame.\n",
    "comparator.pose_camera_once(camera_in_world=[2.25, 0, 2.25],\n",
    "                            target_in_world=[0, 0, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time we've put the camera between the two spheres and the\n",
    "red light in the environment map. We can still clearly see the\n",
    "specular reflection of the white light and the red light in the\n",
    "smooth sphere on the left. The rough sphere on the right has no\n",
    "such specular reflection.\n",
    "\n",
    "But this image illustrates two additional phenomena: Fresnel\n",
    "reflections and diffuse illumination.\n",
    "\n",
    "The Fresnel reflection is apparent in the smooth sphere. We can\n",
    "see the blue light reflected on the rim of the smooth sphere.\n",
    "The fact that the sphere reflects the blue light is expected. But\n",
    "the fact that the reflection of the blue light is more *intense*\n",
    "than the white or red lights might be surprising. The light striking\n",
    "the surface at a grazing angle experiences greater reflectivity\n",
    "so we observe a more intense reflection.\n",
    "\n",
    "Both spheres have a reddish tint that covers a large portion\n",
    "of the side of the sphere. If you move the camera in Meshcat to\n",
    "look at the blue sides of the spheres and re-render, you'll see\n",
    "a similar blue tint on the other side. This is the environment\n",
    "map's contribution to the diffuse lighting effects.\n",
    "\n",
    "Remember, diffuse illumination is the portion of radiant energy that comes\n",
    "from the portion of the light that strikes the surface, refracts inwards,\n",
    "bounces around and, finally is partially emitted again. The\n",
    "_total_ diffuse effect is the sum of all such light. If\n",
    "there's a light source from _any_ direction that can strike a point on the\n",
    "surface, it must be included in the diffuse calculation.\n",
    "\n",
    "In this case, for any given point on the sphere, half of\n",
    "the environment map is able to shine light onto the sphere\n",
    "(it is the hemisphere of the environment centered on the\n",
    "point and arrayed around the surface normal at that point).\n",
    "Because of the integral and the energy-conserving properties\n",
    "of the PBR model, the total contribution of the environment\n",
    "map on the diffuse lighting is a blurred and dimmed version\n",
    "of the environment.\n",
    "\n",
    "This effect can also be thought of as a form of \"ambient\"\n",
    "lighting. When light is shined into a room, the light doesn't\n",
    "simply leave the light source and strike a single surface before\n",
    "bouncing to your eye. The light bounces many, many times. With\n",
    "each bounce the reflected light can change in character. Light\n",
    "simply moves so fast that we can't observe this phenomenon; we\n",
    "merely see the steady state of the light having bounced\n",
    "everywhere. This is the reason why surfaces can be illuminated\n",
    "even if the light source can't shine directly onto the surface;\n",
    "there is \"indirect\" lighting that does shine onto it. This\n",
    "indirect lighting is the \"ambient\" light of the environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### High-Dynamic Range (HDR) Images\n",
    "\n",
    "Not all images can serve equally well as environment maps. A typical image --\n",
    "one typically seen as part of a webpage -- uses \"normalized\" color. That means\n",
    "the value for each pixel is a value in the range [0, 1] (or [0, 255] depending\n",
    "on encoding). The environment map above uses just such an image. Normalized\n",
    "color is sufficient for communicating *basic* color information (hue,\n",
    "saturation, etc.) but it has limited ability to communicate the full _dynamic_\n",
    "range of radiant _energy_ possible. While physical light has varying frequency\n",
    "(signalling color in our minds), it also has _intensity_. An ideal environment\n",
    "map is one that can encode both color _and_ intensity. High-dynamic range (HDR)\n",
    "images use _unnormalized_ color data. They still communicate color properties\n",
    "(again, hue, saturation, etc.), but they also encode intensity. The range\n",
    "between the dimmest and brightest portions of the environment map can be\n",
    "arbitrarily large (as it is in most human environments). We frequently don't\n",
    "notice the wide range of intensities because our eyes and brains are good at\n",
    "taking a wide range of light levels and creating a coherent image in a narrow\n",
    "range of energy levels. Cameras typically have a smaller dynamic range than\n",
    "the human optical system, so photographers are always worrying about shutter\n",
    "speed, film speed, and aperture to control how much light energy the image\n",
    "medium is exposed to.\n",
    "\n",
    "This is of critical importance when it comes to curating environment maps.\n",
    "Environment maps encoded in normalized color images (e.g., png, jpg, bmp, etc.)\n",
    "will have a different impact on the rendering than an unnormalized color image\n",
    "(.hdr). In an environment map, a light intensity is associated with each\n",
    "pixel. Light intensity can be characterized with various units. We'll use\n",
    "watts (W) as being reasonably correlated with the _brightness_ of the light.\n",
    "For a normalized image, the maximum energy at _every_ pixel, regardless of color, is 1 W (pure white pixels); colors with less value (darker colors) implicitly\n",
    "have less energy with pure black being the only zero-energy pixel. To put that in\n",
    "context, in real world scenarios, intensity can range multiple orders of magnitude\n",
    "between the darkest and brightest light sources. That has two implications for\n",
    "normalized color images:\n",
    "\n",
    "  - The maximum intensity of 1 W is quite dim compared to the real world.\n",
    "  - They provide almost _no_ lighting contrast. What contrast exists usually\n",
    "    comes from the _colors_ in the image and the general correlation with low\n",
    "    intensity light sources manifesting as dark colors to film and sensors. As\n",
    "    counter-intuitive as it might seem, a pixel with _dark_ normalized color\n",
    "    (0.01, 0.01, 0.01) is putting out 1 W of light, just like a white\n",
    "    pixel. It's just 1 W of essentially \"black\" light (but not in the \"all-night\n",
    "    rave\" sense of the word)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each environment map encodes a certain amount of radiant energy. Let's take a\n",
    "look at the impact the difference in energy can have.\n",
    "\n",
    "To that end, we'll use three different environment maps which _only_ differ in\n",
    "how much irradiant energy they inject in the scene. All three maps represent\n",
    "the same abstract environment shown above (with the red, blue, and white\n",
    "area lights). They have the same resolution and color information at each pixel.\n",
    "\n",
    "  - PNG: a normalized image with brightness of 1 W.\n",
    "    - The irradiant energy is limited by the file format itself.\n",
    "  - 7.5 HDR: an HDR image with maximum brightness of 7.5 W.\n",
    "    - The irradiant energy in the environment map was controlled by specifying\n",
    "      the power of the corresponding lights in the scene. The brightest light\n",
    "      was a 7.5-watt light.\n",
    "  - 120 HDR: an HDR image with maximum brightness of 120 W.\n",
    "    - The power of all lights in the 7-5-watt scene were multiplied by a\n",
    "      constant factor of 16 (making the brightest light 120 Watts).\n",
    "\n",
    "For the next few experiments, we'll create a small matrix of spheres. \n",
    "Horizontally, we'll vary smoothness, and vertically we'll vary\n",
    "material (metallic vs dielectric)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "# The 7.5 W and 120 W images.\n",
    "package_map = PackageMap()\n",
    "hdr75_path = package_map.ResolveUrl(\n",
    "    \"package://drake_models/tutorials/environment_maps/env_color_room_75.hdr\")\n",
    "hdr1200_path = package_map.ResolveUrl(\n",
    "    \"package://drake_models/tutorials/environment_maps/env_color_room_1200.hdr\")\n",
    "\n",
    "def make_config(env_map_path):\n",
    "    env_texture = EquirectangularMap(path=env_map_path)\n",
    "    return {\"environment_map\": EnvironmentMap(\n",
    "                                   skybox=True,\n",
    "                                   texture=env_texture),\n",
    "            # Disable the light; only environment map light in the scene.\n",
    "            \"lights\": [LightParameter(intensity=0)]}\n",
    "\n",
    "comparator = RenderComparator(meshcat)\n",
    "\n",
    "# Add some geometry to the scene.\n",
    "comparator.add_ground()\n",
    "\n",
    "# The array of spheres.\n",
    "# Bottom row is non-metallic (dielectric), top row is metallic.\n",
    "# Left column is smooth, right column is rough.\n",
    "comparator.add_ball(pbr_material={\"roughnessFactor\": 0.0, \"metallicFactor\": 0},\n",
    "                   position=[-1.1, 0, 1])\n",
    "comparator.add_ball(pbr_material={\"roughnessFactor\": 1.0, \"metallicFactor\": 0},\n",
    "                   position=[1.1, 0, 1])\n",
    "\n",
    "comparator.add_ball(pbr_material={\"roughnessFactor\": 0.0, \"metallicFactor\": 1},\n",
    "                   position=[-1.1, 0, 3.1])\n",
    "comparator.add_ball(pbr_material={\"roughnessFactor\": 1.0, \"metallicFactor\": 1},\n",
    "                   position=[1.1, 0, 3.1])\n",
    "\n",
    "\n",
    "# Make sure we push the geometry to the meshcat browser session.\n",
    "comparator.visualize()\n",
    "\n",
    "comparator.set_render_engines([\n",
    "    RenderEngineSpec(\"Environment Map - 1-watt PNG\", make_config(png_path)),\n",
    "    RenderEngineSpec(\"Environment Map - 7.5-watt HDR\", make_config(hdr75_path)),\n",
    "    RenderEngineSpec(\"Environment Map - 120-watt HDR\", make_config(hdr1200_path)),\n",
    "    ])\n",
    "\n",
    "# Center on the balls, filling the frame.\n",
    "comparator.pose_camera_once(camera_in_world=[0, -3.5, 2],\n",
    "                            target_in_world=[0, 0, 2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although the three images use environment maps with the exact same resolution\n",
    "and the same *color* values at every pixel, the differences in pixel energy\n",
    "produce significantly different results.\n",
    "\n",
    "The 1-watt .png image is both dim and has low contrast (for the reasons\n",
    "suggested above). The 7.5-watt image contains a larger _range_ of\n",
    "illumination and leads to a more vibrant image with greater contrast. The\n",
    "image rendered with the 120-watt image takes this even further.\n",
    "So far, in fact that the image has become oversaturated and the three of the\n",
    "four balls have all but disappeared.\n",
    "\n",
    "Just as we can take a direct light and increase its intensity so much\n",
    "that the rendered image becomes painfully oversaturated, environment maps\n",
    "can do the same.\n",
    "\n",
    "(The appearance of the background is also changing. But, instead of getting\n",
    "lighter like the foreground elements, it's getting darker. We'll look into that\n",
    "in the next section.)\n",
    "\n",
    "The rendered images clearly depend significantly on the light levels in the\n",
    "environment maps. Does this mean that there are some environment maps that are\n",
    "simply unusable? They have too much energy? Good news! We are free to\n",
    "use any environment map; we just need to control how the camera's \"film\" is\n",
    "affected by the light."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"exposure\"></a>\n",
    "#### Tone mapping and exposure\n",
    "\n",
    "We need to take a brief detour to understand how high dynamic range images get\n",
    "drawn to your screen (and saved as normalized color images).\n",
    "\n",
    "In the images above, the image using the 1-watt .png image has 1/120th the light\n",
    "energy as the image rendered with the 120-watt environment map. And yet, the\n",
    "\"dim\" image is not simply a copy of the bright image with each pixel having\n",
    "1/120th the value of the original. If that were the case, the image would just\n",
    "be near-black silhouettes of the balls against a black background. However, the\n",
    "balls are _not_ silhouettes. There is still visible gradation of light\n",
    "communicating the rounded shape of the balls. In fact, the dim, 1-watt image\n",
    "shows more of the spherical shape for the bottom row of balls than the\n",
    "oversaturated 120-watt image. So, what's happening?\n",
    "\n",
    "The answer is \"tone mapping\". It's the process by which an image with\n",
    "\"unbounded\" intensity levels gets mapped to an image (or display) with\n",
    "normalized color. The mapping is not _strictly_ linear.\n",
    "Generally, near the darkest and brightest features, there is non-linear\n",
    "tapering in attempt to preserve details at the extremes of illumination.\n",
    "Furthermore, as the images show above, the tone mapping function has a finite\n",
    "range -- intensity values outside of that range produce the saturated white\n",
    "pixels we've seen in some of the previous images.\n",
    "\n",
    "There are, in fact, various possible mapping functions. \n",
    "`RenderEngineVtk` uses the <a href=\"https://www.kitware.com/pbrj1/\">\"Generic\n",
    "Filmic tone mapping operator with the \"Uncharted2\" parameter values</a>.\n",
    "There are multiple parameters associated with that operator. To keep things\n",
    "simple, Drake exposes a single parameter: `exposure`. If you are familiar\n",
    "with photography, exposure plays a similar role in rendering as it does in\n",
    "photography. It is essentially a way to control how much\n",
    "the \"film\" is exposed to the the scene's radiant energy. Expose it less and the\n",
    "final image is darker. Expose it more, and the image is brighter. Go too far\n",
    "in either direction, and the final image can either be a dark mess or a washed\n",
    "out blur of whiteness. In `RenderEngineVtk`, `exposure` is simply a scale factor\n",
    "applied to the image prior to performing the non-linear mapping. So, if a\n",
    "pixel in the resulting image is reflecting 10 W to the camera, an `exposure`\n",
    "value of 0.5 will mean we map a 5-watt intensity at that pixel. The default\n",
    "value for `exposure` is 1. Images that are too dark should increase exposure\n",
    "and images that are too bright should reduce exposure.\n",
    "\n",
    "It's clear in the above image that the image using the 1-watt .png image is\n",
    "underexposed (three of the four balls are obviously underlit). At the same\n",
    "time, the image using the 120-watt .hdr image is overexposed. Let's render\n",
    "the images again, but this time tune the exposure.\n",
    "\n",
    "The exposure values below, have been selected so that the _top_ of the\n",
    "rough, dielectric sphere basically appears the same in all images. (We'll\n",
    "come back to the question of how do you pick exposure values later.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "def make_config(env_map_path, exposure):\n",
    "    env_texture = EquirectangularMap(path=env_map_path)\n",
    "    return {\"environment_map\": EnvironmentMap(\n",
    "                                   skybox=True,\n",
    "                                   texture=env_texture),\n",
    "            \"exposure\": exposure,\n",
    "            # Disable the light; only environment map light in the scene.\n",
    "            \"lights\": [LightParameter(intensity=0)]}\n",
    "\n",
    "comparator.set_render_engines([\n",
    "    RenderEngineSpec(\"Increased Exposure - 1-watt PNG\",\n",
    "                     make_config(png_path, exposure=7.5)),\n",
    "    RenderEngineSpec(\"Default Exposure - 7.5-watt HDR\",\n",
    "                     make_config(hdr75_path, exposure=1.0)),\n",
    "    RenderEngineSpec(\"Decreased Exposure - 120-watt HDR\",\n",
    "                     make_config(hdr1200_path, exposure=0.0833)),\n",
    "    ])\n",
    "\n",
    "# Center on the balls, filling the frame.\n",
    "comparator.pose_camera_once(camera_in_world=[0, -3.5, 2],\n",
    "                            target_in_world=[0, 0, 2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the effect of exposure on the three images. In all three cases, the\n",
    "*top* of the rough dielectric ball appears about the same.\n",
    "\n",
    "While the target ball in the 1-watt image now resembles the ball in the 7.5-watt\n",
    "image, the other balls and background have changed *significantly*. What happened?\n",
    "In this case, it's because the \"dark\" area of the environment map is not true\n",
    "black. There is a non-zero amount of illumination which has been scaled by a factor\n",
    "of 7.5, pushing it far enough from black that when the subsequent tone mapping is\n",
    "applied, \"black\" has become medium grey. This illustrates an important principle.\n",
    "If the lighting levels lie within a narrow range, exposure *may not* be sufficient\n",
    "to increase contrast.\n",
    "\n",
    "The target ball in the 120-watt image has been pulled back to the same grey as\n",
    "the 7.5-watt image. It is no longer over saturated and the curvature of the ball\n",
    "can easily be seen. However, the black in the mirror ball is _darker_ than the\n",
    "mirror ball in the 7.5-watt image. This is because the 120-watt image has a\n",
    "larger range. However, the grey background in the 7.5-watt and 120-watt images\n",
    "has the same energy level. But in the 120-watt image, that amount is much smaller\n",
    "compared to the maximum intensity. So, when tone mapping is applied, that\n",
    "background intensity is functionally black and renders as such.\n",
    "\n",
    "Judicious use of exposure is critical in producing images that are both\n",
    "aesthetically pleasing as well as modeling sensors which also are sensitive to\n",
    "exposure. That said, remember we've currently only exposed a single parameter\n",
    "related to tone mapping. In the future, we may expose more parameters if\n",
    "exposure proves to be insufficient on its own."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Where do you get environment maps?\n",
    "\n",
    "The two easiest ways to get environment maps are from downloading or\n",
    "creating.\n",
    "\n",
    "Open source environment maps can be downloaded from repositories like\n",
    "[Polyhaven](https://polyhaven.com/).\n",
    "\n",
    "The simple environment maps used in this tutorial were created in Blender.\n",
    "There are copious tutorials on Youtube on how to do this, like\n",
    "[this one](https://www.youtube.com/watch?v=6YMlMEXsiYU)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"total-illumination\"></a>\n",
    "## Total Illumination\n",
    "\n",
    "Environment maps can't cast shadows. Direct lights can. A fully-lit scene\n",
    "can include *both*.\n",
    "\n",
    "Getting them to work together can be tricky. It's easy to configure lighting\n",
    "with both environment map and direct lights such that one of the two lighting\n",
    "sources has no apparent effect.\n",
    "\n",
    "This typically arises if there is a significant difference between the energy\n",
    "levels of the direct lights and the environment map.\n",
    "\n",
    "To get shadows _and_ the benefit of the environment map, it is neccessary to\n",
    "balance the direct light intensity with the environment. (Failing to do so\n",
    "would be like trying to use a flashlight to cast shadows outdoors on a sunny\n",
    "day.)\n",
    "\n",
    "The images below use a fixed exposure, an environment map, and a downward-pointing\n",
    "spot light. The scenes differ in the spot light's intensity. At low intensity,\n",
    "the effect of the light is almost negligible. As the light's intensity increases,\n",
    "the effect of the light becomes apparent and we get effective shadows. As\n",
    "the light becomes too powerful, the image becomes oversaturated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def make_config(intensity):\n",
    "    env_texture = EquirectangularMap(path=hdr75_path)\n",
    "    light = LightParameter(type=\"spot\", frame=\"world\",\n",
    "                           position=[2, -2, 8], cone_angle=40,\n",
    "                           direction=[-0.25, 0.25, -1],\n",
    "                           intensity=intensity)\n",
    "    return {\"environment_map\": EnvironmentMap(\n",
    "                                   skybox=True,\n",
    "                                   texture=env_texture),\n",
    "            \"cast_shadows\": True,\n",
    "            \"shadow_map_size\": 1024,\n",
    "            # Disable the light; only environment map light in the scene.\n",
    "            \"lights\": [light]}\n",
    "\n",
    "engines = []\n",
    "for intensity in [0.1, 1.0, 2.0, 7.5, 100.0]:\n",
    "    engines.append(RenderEngineSpec(f\"Map and {intensity} W light\", make_config(intensity)))\n",
    "comparator.set_render_engines(engines)\n",
    "\n",
    "# Center on the balls, filling the frame.\n",
    "comparator.pose_camera_once(camera_in_world=[0, -3.5, 2],\n",
    "                            target_in_world=[0, 0, 2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we did before with oversaturated images, we'll use `exposure` to reduce\n",
    "the oversaturation effects. And, as before, we'll use the illumination of the\n",
    "rough dielectric ball as the guide for choosing the exposure. In this case,\n",
    "we disabled the shadows while determining those values (as the shadows\n",
    "interfere with our lighting rubric)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def make_config(intensity, exposure):\n",
    "    env_texture = EquirectangularMap(path=hdr75_path)\n",
    "    light = LightParameter(type=\"spot\", frame=\"world\",\n",
    "                           position=[4, -4, 8], cone_angle=40,\n",
    "                           direction=[-4, 4, -8],\n",
    "                           intensity=intensity)\n",
    "    return {\"environment_map\": EnvironmentMap(\n",
    "                                   skybox=True,\n",
    "                                   texture=env_texture),\n",
    "            \"exposure\": exposure,\n",
    "            \"cast_shadows\": True,\n",
    "            \"shadow_map_size\": 1024,\n",
    "            \"lights\": [light]}\n",
    "\n",
    "engines = []\n",
    "for intensity, exposure in [(0.1, 0.95),(1.0, 0.55), (2.0, 0.4), (7.5, 0.11), (100.0, 0.0135)]:\n",
    "    engines.append(RenderEngineSpec(f\"Map and {intensity} W light\", make_config(intensity, exposure)))\n",
    "comparator.set_render_engines(engines)\n",
    "\n",
    "# Center on the balls, filling the frame.\n",
    "comparator.pose_camera_once(camera_in_world=[0, -3.5, 2],\n",
    "                            target_in_world=[0, 0, 2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In all five images, the rough dielectric balls are basically the same.\n",
    "This has understandbly led to attendendant changes in the other balls.\n",
    "\n",
    "There are multiple important lessons to draw from the images above.\n",
    "\n",
    "**Balance**: The images with the 0.1-watt and 100-watt lights show massive\n",
    "imbalance between the direct lights and the environment map. In the 0.1-watt\n",
    "image, the spot light has almost no effect (only the faintest of shadows\n",
    "can be seen below the spheres). In the 100-watt image, the light outshines\n",
    "the environment map by so much, that the perfect specular reflections\n",
    "in the smooth metallic ball have been almost entirely eliminated by the\n",
    "tone mapping. The fundamentally dimmer specular reflections in the smooth\n",
    "dielectric ball _have_ been elimianted.\n",
    "\n",
    "In contrast, the 1-watt and 2-watt (and even the 7.5-watt) images show a\n",
    "better balance between direct illumination and environment map.\n",
    "\n",
    "**Materials**: The mirror ball is not just perfectly smooth (`roughness` = 0),\n",
    "but it is also metallic. Metallic materials are unaffected by diffuse light\n",
    "effects, including from the environment map. The appearance of metallic\n",
    "surfaces are dominated by specular reflection.\n",
    "\n",
    "If the material were non-metallic instead, in addition to the specular\n",
    "reflection attributable to its smoothness, it would also be illuminated by\n",
    "the diffuse and ambient lights.\n",
    "\n",
    "However, the two rough spheres (on the right) appear similar. If metallic\n",
    "balls don't get diffuse light, how is the rough metallic sphere so brightly\n",
    "illuminated? Does the roughness somehow cause diffuse lighting effects?\n",
    "No. It is still only specular reflection, but the roughness has caused the\n",
    "reflection to become so blurred, that it has a diffusive-effect. What you'll\n",
    "notice if you move the camera, is that the appearance of the rough metallic\n",
    "ball will change as the camera moves, whereas the rough dielectric ball\n",
    "remains unchanged.\n",
    "\n",
    "**Exposure values**: The baseline image uses an image consistently described\n",
    "in this tutorial as having a \"7.5-watt white light\". And yet, when we add a\n",
    "1-watt or 2-watt light to the scene, we have to reduce the exposure to 0.55\n",
    "and 0.4, respectively. And when we add a 7.5-watt spot light, the exposure\n",
    "drops all the way down to 0.11 all in the name of normalizing the appearance\n",
    "of the top of the rough sphere.\n",
    "\n",
    "Previously, we said that the exposure is simply a scale factor on the surface's\n",
    "reflected radiant energy. If the environment map has a 7.5-watt top light and\n",
    "we add another 7.5-watt direct light, why would we have to reduce the exposure\n",
    "to ten percent of its original value and not simply half? Conversely, by adding\n",
    "a 1-watt or 2-wat light, we reduced the exposure to around half which implies\n",
    "that the intensity of the environment on the top of the ball was approximately\n",
    "1-2 W.\n",
    "\n",
    "We discussed that the diffuse illumination contributed by the environment\n",
    "map is the result of finding the average light over the hemisphere associated\n",
    "with the surface point and normal. The hemisphere is large and the white\n",
    "light is small (speaking as solid angles). Therefore, the diffuse intensity\n",
    "is significantly lower than the peak intensity in some portion of the hemisphere.\n",
    "If the white light were larger, spanning a larger solid angle, more energy\n",
    "would bleed out as diffuse illumination. In this case, averaging the red, white,\n",
    "and blue light sources with the dark background produces a peak diffuse\n",
    "intensity of only about 1-2 watts from the environment map.\n",
    "\n",
    "This means you can't determine compatible intensities for *direct* lights just by\n",
    "looking at the peak intensities in the environment map.\n",
    "\n",
    "So, how do you pick intensity values for your lighting? For now, tinkering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Putting it all together\n",
    "\n",
    "Let's go ahead and apply all of that in lighting a scene.\n",
    "\n",
    "We'll build up a scene of something more interesting than spheres and then we'll iterate\n",
    "on the lighting in the scene.\n",
    "\n",
    "First, let's populate the world with a table, a robot arm, and a tea set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pydrake.multibody.parsing import (\n",
    "    Parser,\n",
    ")\n",
    "from pydrake.multibody.plant import (\n",
    "    AddMultibodyPlant,\n",
    "    MultibodyPlantConfig,\n",
    ")\n",
    "\n",
    "# We'll use this to load assets from Drake's model repository using package://drake_models URLs.\n",
    "package_map = PackageMap()\n",
    "\n",
    "yaml_str = f\"\"\"\n",
    "directives:\n",
    "- add_model:\n",
    "    name: table\n",
    "    file: package://drake_models/tutorials/models/wooden_table.sdf  \n",
    "- add_weld:\n",
    "    parent: world\n",
    "    child: table::table\n",
    "- add_model:\n",
    "    name: iiwa\n",
    "    file: package://drake_models/iiwa_description/urdf/iiwa14_primitive_collision.urdf\n",
    "    default_joint_positions:\n",
    "      iiwa_joint_1: [1.13]\n",
    "      iiwa_joint_2: [0.79]\n",
    "      iiwa_joint_3: [0.03]\n",
    "      iiwa_joint_4: [-2.09]\n",
    "      iiwa_joint_5: [-1.19]\n",
    "      iiwa_joint_6: [1.16]\n",
    "      iiwa_joint_7: [-0.3]\n",
    "- add_weld:\n",
    "    parent: table_top_center\n",
    "    child: iiwa::base\n",
    "    X_PC:\n",
    "      translation: [-0.3, 0, 0.0]\n",
    "- add_model:\n",
    "    name: wsg\n",
    "    file: package://drake_models/wsg_50_description/sdf/schunk_wsg_50_with_tip.sdf\n",
    "    default_joint_positions:\n",
    "      left_finger_sliding_joint: [-0.02]\n",
    "      right_finger_sliding_joint: [0.02]\n",
    "- add_frame:\n",
    "    name: wsg_on_iiwa\n",
    "    X_PF:\n",
    "      base_frame: iiwa_link_7\n",
    "      translation: [0, 0, 0.114]\n",
    "      rotation: !Rpy {{ deg: [90, 0, 90] }}\n",
    "- add_weld:\n",
    "    parent: wsg_on_iiwa\n",
    "    child: wsg::body  \n",
    "- add_model:\n",
    "    name: tea_set\n",
    "    file: package://drake_models/tutorials/models/tea_set.sdf\n",
    "- add_weld:\n",
    "    parent: table::table_top_center\n",
    "    child: tea_set::tea_set\n",
    "    X_PC:\n",
    "      translation: [0.3, -0.05, 0]\n",
    "\"\"\"\n",
    "\n",
    "builder = DiagramBuilder()\n",
    "plant, scene_graph = AddMultibodyPlant(config=MultibodyPlantConfig(),\n",
    "                                       builder=builder)\n",
    "parser = Parser(plant)\n",
    "parser.AddModelsFromString(yaml_str, \".dmd.yaml\")\n",
    "plant.Finalize()\n",
    "\n",
    "# Configure the canonical camera pose.\n",
    "p_WC = [0.87435, 0.52266, 1.21673]\n",
    "p_WT = [0.03554, 0.07906, 0.90111]\n",
    "\n",
    "comparator = RenderComparator(meshcat, builder)\n",
    "\n",
    "# An accumulator for the render engines we're going to compare.\n",
    "engines = []\n",
    "\n",
    "def add_engine_spec(engine_spec, engine_list = None, limit = 0):\n",
    "    \"\"\"Adds the given engine_spec to engines. If an engine with that\n",
    "    name already exists, it gets replaced.\"\"\"\n",
    "    global engines\n",
    "    engine_list = engines if engine_list is None else engine_list\n",
    "    for i in range(len(engine_list)):\n",
    "        if engine_list[i].label == engine_spec.label:\n",
    "            engine_list[i] = engine_spec\n",
    "            return\n",
    "    engine_list.append(engine_spec)\n",
    "    if (limit > 0):\n",
    "        while len(engine_list) > limit:\n",
    "            engine_list.pop(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get ready to render.\n",
    "\n",
    "  - We're going to change the camera a bit. Rather than using\n",
    "    Meshcat's camera, we'll more closely approximate a 35-mm\n",
    "    lens by changing the focal length.\n",
    "  - We'll add a default RenderEngineVtk. It *will* automatically\n",
    "    be in PBR mode because all of the models we're adding are\n",
    "    glTF files.\n",
    "  - Finally, we'll pick an initial camera pose so we can see\n",
    "    what's happening.\n",
    "    \n",
    " Evaluating the next block *may* print a warning. You can ignore it.\n",
    " Drake's models include texture optimizations for Meshcat that\n",
    " VTK ignores. We may see that same warning each time after we call\n",
    " `comparator.set_render_engines()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Vertical field of view for a 35 mm camera with a 4:3 aspect ratio.\n",
    "comparator.set_camera_parameters({\"fov_y\": 40.8 * np.pi / 180})\n",
    "default_spec = RenderEngineSpec(\"All Defaults\", {})\n",
    "add_engine_spec(default_spec)\n",
    "comparator.set_render_engines(engines)\n",
    "comparator.pose_camera_once(camera_in_world=p_WC, target_in_world=p_WT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's render it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can clearly see the table with the robot arm and tea set on top. The gripper appears to be reaching for the teapot's handle. We made no explicit declarations about the lighting, so we have the default \"headlamp\" light for `RenderEngineVtk`.\n",
    "\n",
    "The background color is quite light and the scene objects appear relatively dark. For now, we won't worry about the background color (we'll address that later). But let's tweak the exposure so that we can make out the objects in our scene better. The `exposure` value used below is completely subjective; we picked a value such that we liked the rendered result more than the other values we sampled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bright_default_spec = RenderEngineSpec(\"Default (Exposure = 4)\", {\"exposure\": 4})\n",
    "add_engine_spec(bright_default_spec)\n",
    "comparator.set_render_engines(engines)\n",
    "comparator.pose_camera_once(camera_in_world=p_WC, target_in_world=p_WT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The scene objects are now much easier to see.\n",
    "\n",
    "It's a good starting point, but we don't necessarily want the objects floating in the clouds. So, let's give it an environment. Using models available through [Drake's model repository](https://github.com/RobotLocomotion/models), we've put together a simple environment and rendered it into an environment map. This is like what we did above to create our toy environment map, but this time, it's a bit more interesting.\n",
    "\n",
    "Let's load the environment and take a look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "map_name = f\"package://drake_models/tutorials/environment_maps/env_home_office_full_lights\"\n",
    "# The HDR map we'll use in the rendering.\n",
    "hdr_path = package_map.ResolveUrl(map_name + \".hdr\")\n",
    "# The png version we'll use to draw here and send to Meshcat.\n",
    "png_path = package_map.ResolveUrl(map_name + \".png\")\n",
    "\n",
    "plt.imshow(ImageIo().Load(png_path).data)\n",
    "\n",
    "# Also send the environment map to the meshcat session.\n",
    "meshcat.SetEnvironmentMap(png_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The room has four lights on the ceiling and a window open to the daytime sky. The light coming through the window is brighter than that from the light bulbs.\n",
    "\n",
    "Take a look at the environment map in the meshcat visualizer to get a better view of the room.\n",
    "\n",
    "Now let's apply the environment map to the scene."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "invisible_env_spec = RenderEngineSpec(\n",
    "    \"Invisible Environment Map\",\n",
    "    {\n",
    "        \"environment_map\": EnvironmentMap(\n",
    "            skybox=False, texture=EquirectangularMap(path=hdr_path)),\n",
    "        \"exposure\": 2,\n",
    "        \"lights\": [LightParameter(intensity=0)]\n",
    "    }\n",
    ")\n",
    "add_engine_spec(invisible_env_spec)\n",
    "comparator.set_render_engines(engines)\n",
    "comparator.pose_camera_once(camera_in_world=p_WC, target_in_world=p_WT)\n",
    "meshcat.SetEnvironmentMap(png_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: we're not rendering the environment map in the background (`skybox=False`). There are several notable differences between the images:\n",
    "\n",
    "  - Reducing the exposure has unsurprisingly caused the background color to get darker.\n",
    "  - There are clearly more apparent light sources; you can see multiple highlights on the robot arm.\n",
    "  - Despite the reduced exposure, the table is more brightly illuminated in this image than it is in the previous image. This is due to several reasons:\n",
    "     - In the previous image, the camera is facing forward in the camera view, so the light direction is moving mostly *across* the table surface instead of down onto it. In contrast, the environment map has multiple lights shining from above the table.\n",
    "     - The intensity of the lights in the environment map are greater than the single default headlamp.\n",
    "  - The light positions are independent of the camera position. The means much of the robot arm appears darker because it's no longer being illuminated directly from the camera position.\n",
    "     - In fact, if you turn the camera in meshcat to look at the scene from the window's side, you'll be able to see the window's contribution to the scene.\n",
    "\n",
    "The addition of the environment map is a definite improvement over the default light.\n",
    "\n",
    "Let's see what it looks like when we render the environment map."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visible_env_map = RenderEngineSpec(\n",
    "    \"Visible Environment Map\",\n",
    "    {\n",
    "        \"environment_map\": EnvironmentMap(\n",
    "            skybox=True, texture=EquirectangularMap(path=hdr_path)),\n",
    "        \"exposure\": 2,\n",
    "        \"lights\": [LightParameter(intensity=0)]\n",
    "    }\n",
    ")\n",
    "add_engine_spec(visible_env_map)\n",
    "comparator.set_render_engines(engines)\n",
    "comparator.pose_camera_once(camera_in_world=p_WC, target_in_world=p_WT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The foggy background is now gone. Try moving the meshcat camera and re-render the scene. Try some of the following angles:\n",
    "\n",
    "  - Look at the table across from the window (so that you can see the window in the rendering).\n",
    "  - Do the same, but face the room's door.\n",
    "  - Look from the perspective of the window.\n",
    "  \n",
    "In two of those views, the background seems to be overly dark. However, when looking at the robot arm and the window in the same rendering, the background and scene elements seem far more well balanced. Why is that?\n",
    "\n",
    "It's primarily due to the light intensity. Before we apply tone mapping (via the `exposure` parameter), we first collect the amount of light energy at each pixel. When we put the camera at the window and look at the table, we see the window's *full* light contribution reflecting off the scene's objects. *Behind* those objects, however, we have a wall which contributes very little light in comparison. Therefore, when we apply the tone mapping, the brightest part of the image (the foreground elements) look just right, and the background gets pushed closer to black.\n",
    "\n",
    "When we look at the table with the window *behind* the table, both the window's light energy and the highlights on the table, robot, etc. have a similar intensity. So, they get treated in a very similar way by the tone mapping process.\n",
    "\n",
    "This is an inherent problem with environment maps. Their *true* purpose is to serve as a source of lighting to the objects in the scene. Also using it as a backdrop in the scene may not have the desired result. So, what can be done?\n",
    "\n",
    "The simplest solution is to actually put in a background; objects that can actually reflect the same light as the foreground elements.\n",
    "\n",
    "Let's reload the scene, but this time, we'll add the geometry of the room as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "builder = DiagramBuilder()\n",
    "plant, scene_graph = AddMultibodyPlant(config=MultibodyPlantConfig(),\n",
    "                                       builder=builder)\n",
    "parser = Parser(plant)\n",
    "room_yaml = \"\"\"\n",
    "- add_directives:\n",
    "    file: package://drake_models/tutorials/models/furnished_living_room.dmd.yaml  \n",
    "\"\"\"\n",
    "parser.AddModelsFromString(yaml_str + room_yaml, \".dmd.yaml\")\n",
    "plant.Finalize()\n",
    "\n",
    "comparator = RenderComparator(meshcat, builder)\n",
    "\n",
    "# Vertical field of view for a 35 mm camera with a 4:3 aspect ratio.\n",
    "comparator.set_camera_parameters({\"fov_y\": 40.8 * np.pi / 180})\n",
    "\n",
    "# Also cull the unneeded renderings; we don't need the first default engine\n",
    "# or the last engine with the skybox.\n",
    "engines = engines[1:-1]\n",
    "\n",
    "comparator.set_render_engines(engines)\n",
    "comparator.pose_camera_once(camera_in_world=p_WC, target_in_world=p_WT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The inclusion of the background geometry has two immediate impacts:\n",
    "\n",
    "  - There is something to reflect the same light that the robot is; the background is lighter than when we were simply using the environment map as a backdrop.\n",
    "  - The perspective on the background is correct.\n",
    "    - Environment maps treat the environment like it's infinitely far away from the illuminated objects.\n",
    "    - In this case, that's a very poor approximation of the environment; the room's walls are just a few meters away from the foreground elements. When the camera moves away from the center of the environment's center, what is shown in the background is increasingly incorrect. By including geometry for the environment, it maintains the correct spatial relationship as the camera moves.\n",
    "    \n",
    "**Note** There is currently a known defect in `RenderEngineVtk`; transparent objects are not rendered properly. So, when you turn the camera to look across the robot into the window, it is no longer brightly illuminated; the window is treated as opaque."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the environment map correlates with the room geometry, we're getting great specular and diffuse effects which really tie the geometry in with the rest of the room. Let's see if we can get some shadows.\n",
    "\n",
    "When adding lights, we advocate the following process:\n",
    "\n",
    "0. Pull the camera back so you can actually see what you're doing.\n",
    "1. Add one light at a time.\n",
    "2. When adding a light, make sure it's the only light source.\n",
    "3. Set the light intensity incredibly high so you can see its effect. This will help getting the posing of the light right; you want to easily see where the light is being cast.\n",
    "4. Disable shadows; an occluded light provides very few clues on why it isn't contributing.\n",
    "5. After the pose is confirmed:\n",
    "    - Enable shadows.\n",
    "    - Reintroduce other light sources.\n",
    "    - Tune the new light's properties to work with the existing set.\n",
    "\n",
    "So, let's do that. Let's attempt to create shadows from the open window. In this case, we know the window lies in the -Wz direction from the robot arm. So, we're going to start the light shining in the +Wy direction. We'll simply run through a number of iterations in sequence without discussion (beyond code comments)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First attempt:\n",
    "#   1. Omit the environment map.\n",
    "#   2. Directional light shining in the +Wy direction.\n",
    "#   3. Set intensity high.\n",
    "env = RenderEngineSpec(\n",
    "    \"Iteration 1\",\n",
    "    {\n",
    "        \"lights\": [\n",
    "               LightParameter(type=\"directional\", frame=\"world\", direction=[0, 1, 0], intensity=10),\n",
    "        ],\n",
    "    }\n",
    ")\n",
    "lighting_engines = []\n",
    "add_engine_spec(env, lighting_engines)\n",
    "comparator.set_render_engines(lighting_engines)\n",
    "# Pull the camera back.\n",
    "comparator.pose_camera_once(camera_in_world=[2.62940, -2.85036, 2.80655],\n",
    "                            target_in_world=[2.03886, -2.16774, 2.37608])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Second attempt - tilt the light downwards.\n",
    "env = RenderEngineSpec(\n",
    "    \"Iteration 2\",\n",
    "    {\n",
    "        \"lights\": [\n",
    "               LightParameter(type=\"directional\", frame=\"world\", direction=[0, 1, -.5], intensity=10),\n",
    "        ],\n",
    "    }\n",
    ")\n",
    "add_engine_spec(env, lighting_engines)\n",
    "comparator.set_render_engines(lighting_engines)\n",
    "# Pull the camera back.\n",
    "comparator.pose_camera_once(camera_in_world=[2.62940, -2.85036, 2.80655],\n",
    "                            target_in_world=[2.03886, -2.16774, 2.37608])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Third attempt; test shadows.\n",
    "env = RenderEngineSpec(\n",
    "    \"Iteration 3\",\n",
    "    {\n",
    "        \"cast_shadows\": True,\n",
    "        \"lights\": [\n",
    "               LightParameter(type=\"directional\", frame=\"world\", direction=[0, 1, -.5], intensity=10),\n",
    "        ],\n",
    "    }\n",
    ")\n",
    "add_engine_spec(env, lighting_engines)\n",
    "comparator.set_render_engines(lighting_engines)\n",
    "# Pull the camera back.\n",
    "comparator.pose_camera_once(camera_in_world=[2.62940, -2.85036, 2.80655],\n",
    "                            target_in_world=[2.03886, -2.16774, 2.37608])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fourth attempt; put camera back and bring environment map back.\n",
    "env = RenderEngineSpec(\n",
    "    \"Iteration 4\",\n",
    "    {\n",
    "        \"environment_map\": EnvironmentMap(texture=EquirectangularMap(path=hdr_path)),\n",
    "        \"cast_shadows\": True,\n",
    "        \"lights\": [\n",
    "               LightParameter(type=\"directional\", frame=\"world\", direction=[0, 1, -.5], intensity=10),\n",
    "        ],\n",
    "    }\n",
    ")\n",
    "add_engine_spec(env, lighting_engines)\n",
    "comparator.set_render_engines(lighting_engines)\n",
    "# Restore camera.\n",
    "comparator.pose_camera_once(camera_in_world=p_WC, target_in_world=p_WT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Final configuration: Tone down the window light; we only want faint shadows from the window.\n",
    "env = RenderEngineSpec(\n",
    "    \"Iteration 5\",\n",
    "    {\n",
    "        \"environment_map\": EnvironmentMap(texture=EquirectangularMap(path=hdr_path)),\n",
    "        \"cast_shadows\": True,\n",
    "        \"exposure\": 3,\n",
    "        \"lights\": [\n",
    "               LightParameter(type=\"directional\", frame=\"world\", direction=[0, 1, -.5], intensity=1),\n",
    "        ],\n",
    "    }\n",
    ")\n",
    "add_engine_spec(env, lighting_engines, limit=4)\n",
    "comparator.set_render_engines(lighting_engines)\n",
    "comparator.pose_camera_once(camera_in_world=p_WC, target_in_world=p_WT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, now we have some subtle shadows coming in from the window. Let's see if we can't get shadows from the four lights in the ceiling. Applying the same process as above, we arrive at the following. Feel free to try the same process above to determine your own configuration for the ceiling lights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ceiling_light = {\"type\": \"spot\",\n",
    "                 \"frame\": \"world\",\n",
    "                 \"direction\": [0, 0, -1],\n",
    "                 \"cone_angle\": 60,\n",
    "                 \"intensity\": 5,\n",
    "                 \"attenuation_values\": [0, 0, 1]}\n",
    "render_config = {\n",
    "   \"environment_map\": EnvironmentMap(texture=EquirectangularMap(path=hdr_path)),\n",
    "    \"cast_shadows\": True,\n",
    "    \"shadow_map_size\": 1024,\n",
    "    # With additional lights, we reduce exposure.\n",
    "    \"exposure\": 2,\n",
    "    \"lights\": [\n",
    "               LightParameter(**(ceiling_light | {\"position\": [-1, 1, 3.0]})),\n",
    "               LightParameter(**(ceiling_light | {\"position\": [-1, -1, 3.0]})),\n",
    "               LightParameter(**(ceiling_light | {\"position\": [1, 1, 3.0]})),\n",
    "               LightParameter(**(ceiling_light | {\"position\": [1, -1, 3.0]})),\n",
    "               LightParameter(type=\"directional\", frame=\"world\", direction=[0, 1, -.5], intensity=1),\n",
    "              ],\n",
    "}\n",
    "env = RenderEngineSpec(\"Full Lighting\", render_config)\n",
    "add_engine_spec(env, lighting_engines, limit=4)\n",
    "comparator.set_render_engines(lighting_engines)\n",
    "comparator.pose_camera_once(camera_in_world=p_WC, target_in_world=p_WT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "comparator.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The shadows are subtle. They become more apparent when objects are in motion. Sometimes it can be easy to be fooled by a single image.\n",
    "\n",
    "Here are a couple more curated viewpoints to look at the scene."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The example rendering from the top of the tutorial.\n",
    "\n",
    "comparator.set_render_engines(lighting_engines[-1:])\n",
    "comparator.pose_camera_once(camera_in_world=[1.22219, 0.71093, 1.44518],\n",
    "                            target_in_world=[0.45264, 0.18018, 1.09009])\n",
    "comparator.render()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator.set_render_engines(lighting_engines[-1:])\n",
    "comparator.pose_camera_once(camera_in_world=[1.69815, -1.52771, 1.91255],\n",
    "                            target_in_world=[1.04723, -0.87488, 1.52511])\n",
    "comparator.render()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
